Abstract: In the field of medicine, in order to diagnose a patient’s condition more efficiently and conveniently, image classification has been widely leveraged. It is well established that when doctors diagnose a patient’s condition, they not only observe the patient’s image information (such as CT image) but also make final decisions incorporating the patient’s clinical diagnostic information. However, current medical image classification only puts the image into a convolution neural network to obtain the diagnostic result and does not use the clinical diagnosis information. In intelligent auxiliary diagnosis, it is necessary to combine clinical symptoms with other imaging data for comprehensive diagnosis. This paper presented a new method of assistant diagnosis for the medical field. This method combined information from patients’ imaging with numerical data (such as clinical diagnosis information) and used the combined information to automatically predict the patient’s condition. Based on this method, a medical assistant diagnosis model based on deep learning was proposed. The model takes images and numerical data as input and outputs the patient’s condition. Thus, this method is comprehensive and helps improve the accuracy of automatic diagnosis and reduce diagnostic error. Moreover, the proposed model can simultaneously process multiple types of data, thus saving diagnosis time. The effectiveness of the proposed method was verified in two groups of experiments designed in this paper. The first group of experiments shows that if the unrelated data are fused for classification, the proposed method cannot enhance the classification ability of the model, although it is able to predict multiple diseases at one time. The second group of experiments show that the proposed method could significantly improve classification results if the interrelated data are fused.