CN112890815A - Autism auxiliary evaluation system and method based on deep learning - Google Patents

Abstract

The invention provides an autism auxiliary evaluation system and method based on deep learning. The system comprises a data acquisition and feature extraction unit, a first neural network, a second neural network, a third neural network and a result output unit, wherein the data acquisition and feature extraction unit and the result output unit are respectively connected with the first neural network, the second neural network and the third neural network, and the data acquisition and feature extraction unit is used for acquiring eye movement data of a subject watching a video to obtain a heat spot diagram, a focus diagram and a scanning path diagram; inputting a hot spot diagram by a first neural network to obtain a first classification result; the second neural network inputs a focus map to obtain a second classification result; inputting a scanning path diagram by a third neural network to obtain a third classification result; the result output unit collects the first classification result, the second classification result and the third classification result to obtain the autism detection result of the subject. The invention improves the prediction efficiency and the prediction accuracy of the autism.

Description

Autism auxiliary evaluation system and method based on deep learning

Technical Field

The invention relates to the technical field of autism assessment, in particular to an autism auxiliary assessment system and method based on deep learning.

Background

Autism Spectrum Disorder (ASD), or "autism spectrum disorder", "autism" is a heterogeneous neurodevelopmental disorder with social dysfunction and stereotypical behavior as core disorders. The incidence of ASD in children is increasing year by year worldwide, and now becomes a social public health problem. According to the estimates of autism and dysgenesis monitoring networks under the U.S. centers for disease control and prevention, autism is present in 1 out of 68 individuals in the united states, and thus the present day society's concern for ASD has risen dramatically over the past few years. Rehabilitation training is the primary treatment modality for ASD children, and more examples show that the earlier the intervention is initiated, the better the prognosis. By early diagnosis and intervention of the ASD children, the symptoms of the ASD children are relieved, the potential of the ASD children can be exerted to the maximum extent, the corresponding functional level of the ASD children is improved, and the ASD children can thrive in the same health as normal children.

However, the diagnosis of ASD lacks reliable biochemical diagnostic markers or imaging bases, and has great heterogeneity. The ASD has different social communication and interactive disorder damage degrees of core symptoms, the ASD is diversified in performance, particularly, the timely identification and correct diagnosis of the light-moderate ASD children have certain difficulty, diagnosis omission and misdiagnosis are easy, the ASD children are labeled with 'problem children' after entering a mainstream young-holding organization, the ASD children are easy to be subjected to the responsibility of parents, the responsibility and penalty of teachers and the separation of classmates, behavior and emotion problems are caused, and the ASD is very unfavorable for the prognosis of the children. Therefore, a method with convenient use and high reliability is urgently needed for auxiliary diagnosis, thereby being beneficial to early correct diagnosis and timely intervention and lightening the increasingly heavy recovery cost burden of the autism spectrum disorder sick children.

The diagnostic criteria for children with ASD are mainly the "handbook of diagnosis and statistics of mental disorders" of the American psychiatric society and the diagnostic criteria for mental and behavioral disorders of the world health organization. The current mainstream evaluation and prediction method is to diagnose the autism by combining the usual growth and development history, the medical history and the mental examination of children with questionnaire survey and interview by using the internationally mainstream diagnosis tools (an autism diagnosis observation table and an autism diagnosis interview table) according to the diagnosis standard and aiming at three aspects of language communication disorder, social communication disorder and repeated stereotypy behavior. Analysis of ASD visual characteristics is of particular interest around the ASD core-specific symptom, social disturbance, from a social-cognitive perspective, when direct or indirect observations based on behavioral signs and symptoms are no longer applicable. Recent studies provide evidence of different eye movement patterns of ASD individuals, and how ASD individuals scan faces is studied by eye tracking techniques. These studies agree that autistic individuals have less visual attention to the face than normal children. The field of medicine has now begun to study the gaze behavior of the eye as one of the diagnostic criteria for autism in children. From the angle, a type of eye gaze behavior tracking ASD by means of an eye tracker is formed, and evaluation and prediction of autism are performed by analyzing and feature extracting eye movement data of the ASD. On the other hand, with the development of the brain function imaging technology research, a method for diagnosing ASD partially by means of the brain function imaging technology has appeared. The method comprises the steps of obtaining resting state functional magnetic resonance data through a nuclear magnetic resonance scanner or electroencephalogram signals acquired by electroencephalogram equipment, preprocessing the acquired data and carrying out corresponding functional processing to obtain required characteristics for training a classification model to classify autism and normal children. Furthermore, more and more research is being conducted to extract relevant features by means of engineering techniques from multimodal signals from various aspects for the assessment and prediction of autism. For example, the autism tester watches the face image corresponding to the pre-made video and the face temperature change, heart rate change and breathing change when watching the image; or the visual camera is used for assisting in judging the response of the subject to the language; or collecting multi-channel audio/video multi-mode data of multiple RGB-D camera visual angles of a testee, an evaluator and a prop in the laugh test process; or the electroencephalogram signal, the electromyogram signal, the electro-oculogram signal, the skin electrical reaction signal, the body temperature data, the respiratory frequency and the like of the testee in different emotional states are extracted based on the physiological signal, and even the expression characteristics of the testee are directly extracted by utilizing the expression reaction of the testee to perform auxiliary judgment. In summary, a variety of available multi-modal signatures and features have been tried to aid in the diagnosis of autistic children.

In the conventional classification-aided diagnosis method for autism using eye movement data of ASD, the eye movement data or eye gaze data of the observed person is usually acquired by means of an eye tracker or a spectacle-type eye tracker, mainly by viewing a static face picture or a static picture in a visual follow-up task. And performing complicated artificial feature extraction work with the assistance of training of a traditional machine learning classification model Support Vector Machine (SVM) or a BP neural network to obtain a classifier model with better performance, and classifying and identifying the children according to the established classification model. In the scheme of auxiliary diagnosis by using an eye tracking technology, simple static face recognition is mostly taken as a main research direction, and few reports are provided for realizing auxiliary diagnosis of an ASD child patient based on facial emotion recognition. Face recognition and emotion perception disorders are core problems of ASD children social disorders, and facial emotion recognition defects commonly existing in patients with autism spectrum disorders are core causes of social and communication disorders. The patients with the autism spectrum disorder have significant difference from the eye movement patterns of typical development populations under social and facial stimulation conditions, the fixation of the patients with the autism spectrum disorder on the eye areas is less, and the avoidance of direct fixation can be the reason for the development of the facial emotion recognition defect of the patients with the autism spectrum disorder. The current research on the facial emotion recognition disorder of patients with autism spectrum disorder is mainly an eye tracking technology. Studies have shown that ASD children have patterns of eye movements in facial recognition and emotional perception that are different from normal children, a feature that exists in ASD children with symptoms ranging from mild to severe. The specific facial processing patterns of autism spectrum disorder patients are primarily to select for facial stimuli and extract information that is suboptimal for the task of emotion recognition.

In summary, the following problems mainly exist in the existing technical solutions: 1) eye movement-based solutions stimulate material to be too classical and disjointed from real life scenarios. In the current scheme based on eye movement auxiliary diagnosis, the provided stimulation materials are static pictures, are disjointed with actual life scenes, and cannot truly evaluate the emotion recognition capability of the ASD children in the real-world interaction. 2) And is not suitable for auxiliary diagnosis of children in the age range of 6-18 months. The current autism spectrum disorder scale has an age range, and for autism patients, the problem that the problem is difficult to solve or the age is not in the scale evaluation range often occurs. Moreover, for children with younger ages, questionnaires cannot be investigated and interviewed, and diagnosis can be performed only by daily observation of parents and interviews and behavior verification of doctors, so that missed diagnosis and misdiagnosis are easily caused. The existing method for acquiring eye movement data by using a glasses type eyeball tracker and acquiring data by using multi-mode data such as electroencephalogram and the like and the method for using expression reaction are too difficult to realize for children of 6-18 months of age. 3) And cumbersome manual feature extraction work is required. According to the related engineering technical scheme for ASD auxiliary diagnosis at present, no matter an eye movement tracking technology is adopted to obtain eye movement data of a subject, or facial features of the subject in a face recognition related task are obtained, even multi-mode signals such as electroencephalogram signals and electromyogram signals of the subject are obtained, diagnosis and classification are carried out through training of a classifier module after complicated artificial feature extraction work. The obtained data with different forms has to be subjected to elaborate and complicated feature extraction to serve the high-accuracy classification performance in the later period.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provide a deep learning-based autism auxiliary assessment system and method, which combines the eye movement technology and the deep learning to predict and assess autism.

According to a first aspect of the invention, a deep learning-based autism aided assessment system is provided. The system comprises a data acquisition and feature extraction unit, a first neural network, a second neural network, a third neural network and a result output unit, wherein the data acquisition and feature extraction unit and the result output unit are respectively in communication connection with the first neural network, the second neural network and the third neural network, and the data acquisition and feature extraction unit and the result output unit are respectively in communication connection with the first neural network, the second neural network and the third neural network, wherein: the data acquisition and feature extraction unit is used for acquiring eye movement data of a video watched by a subject to obtain a corresponding hotspot graph, a focus graph and a scanning path graph, wherein the hotspot graph is used for representing the dynamic change of the time and the position of a gazing point, the focus graph is used for representing the dynamic change of the gazing position and the time, and the path scanning graph continuously displays the information of the gazing point position and each gazing time point by point; the first neural network is used for inputting the heat point diagram to obtain a first classification result; the second neural network is used for inputting the focus map to obtain a second classification result; the third neural network is used for inputting the scanning path diagram to obtain a third classification result; the result output unit collects the first classification result, the second classification result and the third classification result to obtain the autism detection result of the subject.

In one embodiment, the first, second and third neural networks have the same or different structures.

In one embodiment, the first, second and third neural networks have the same structure, including an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth pooling layer, a fifth convolutional layer, a sixth fully-connected layer, a seventh fully-connected layer and an output layer.

In one embodiment, the activation functions of the first, second, third, fourth, fifth, and sixth fully-connected layers of the first, second, and third neural networks are ReLU nonlinear activation functions, the activation function of the seventh fully-connected layer is Softmax activation function, the number of neurons in the output layer is 4, and the four categories correspond to healthy, mild autistic symptoms, moderate autistic symptoms, and severe autistic symptoms, respectively.

In one embodiment, the result output unit combines the first classification result, the second classification result, and the third classification result using a simple voting method to give a final prediction result.

In one embodiment, the eye movement data for each subject is collected in a non-invasive manner using an eye tracker.

In one embodiment, the heat map displays the dynamic change of time and position of the fixation point in warm color and the focus map displays the dynamic change of fixation position and time in brightness.

According to a second aspect of the invention, a method for assisted assessment of autism based on deep learning is provided. The method comprises the following steps:

the method comprises the steps of collecting eye movement data of a video watched by a subject to obtain a corresponding hotspot graph, a focus graph and a scanning path graph, wherein the hotspot graph is used for representing the dynamic change of time and position of a fixation point, the focus graph is used for representing the dynamic change of the fixation position and time, and the path scanning graph continuously displays the fixation point position and each fixation time information point by point;

inputting the distribution of the heat point diagram, the focus diagram and the scanning path diagram into a trained first neural network, a trained second neural network and a trained third neural network to respectively obtain a first classification result, a second classification result and a third classification result;

and aggregating the first classification result, the second classification result and the third classification result to obtain the autism detection result of the subject.

Compared with the prior art, the invention has the advantages that: starting from the ubiquitous facial emotion recognition defect of an ASD patient, aiming at the research target of ASD eye movement screening, compared with static picture stimulation, a dynamic scene with daily speech expression is adopted as a stimulation material, and emotion recognition reaction and eye movement data of a subject in real social interaction are extracted through stimulation design of a dynamic video, so that the real reliability of auxiliary diagnosis is improved; the eye movement technology is non-invasive, the testee does not need to wear any device, and meanwhile, the stimulation material can be properly adjusted according to the age group of the testee, so that the device is suitable for autism patients of different age groups and different development levels, and is particularly suitable for children of 6-18 months and has a wider application range; according to the existing method for realizing auxiliary diagnosis by using the eye movement technical scheme, most of results are predicted by performing complicated manual feature extraction work and then performing traditional machine learning classification model training, and time and labor are consumed.

Drawings

The invention is illustrated and described only by way of example and not by way of limitation in the scope of the invention as set forth in the following drawings, in which:

FIG. 1 is a schematic diagram of a deep learning based autism assistance system according to one embodiment of the invention;

FIG. 2 is a diagram of a neural network architecture, according to one embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions, design methods, and advantages of the present invention more apparent, the present invention will be further described in detail by specific embodiments with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

The defect of facial emotion recognition commonly existing in patients with autism spectrum disorder is a core cause of social and communication disorders, and currently, eye tracking technology is mainly used for research on facial emotion recognition disorder of the patients with autism spectrum disorder. The patients with the autism spectrum disorder have significant difference from the eye movement patterns of typical development populations under social and facial stimulation conditions, the fixation of the patients with the autism spectrum disorder on the eye areas is less, and the direct fixation is avoided to be the reason for the development of the facial emotion recognition defect of the patients with the autism spectrum disorder. There is evidence that the high level of arousal associated with direct fixation in patients with autism spectrum disorders is associated with the avoidance of direct fixation and with a more severe impairment of social competence. The specific facial processing patterns of autism spectrum disorder patients are primarily to select for facial stimuli and extract information that is suboptimal for the task of emotion recognition.

The invention provides an efficient, convenient, noninvasive and low-cost ASD auxiliary diagnosis method facing face recognition and emotion perception tasks based on an eye movement technology according to the association between eye movement data characteristics and autism spectrum disorder patients. Briefly, an embodiment of the invention includes the steps of: guiding the subject to complete a video watching task in the data acquisition process, requiring the subject to concentrate on the content heard and seen, and recording the eye movement data of the subject by using an eye tracker when the subject watches the video short film; finishing data preprocessing work aiming at the obtained eye image data; then, automatic feature extraction is carried out by combining a Convolutional Neural Network (CNN) in a deep learning method, a Neural network classifier is obtained through model training, and finally, the ASD auxiliary diagnosis is realized. The invention combines the eye tracking data and the deep learning algorithm, and can effectively extract the specific facial processing mode of the autism patient, thereby realizing the auxiliary diagnosis of the mild and moderate autism patient.

Specifically, referring to fig. 1, the assisted evaluation system for autism based on deep learning according to the embodiment of the present invention includes a data acquisition and feature extraction unit 110, a neural network for classification training, and a result output unit 120, which are connected in sequence, where the neural network 1, the neural network 2, and the neural network 3 are illustrated.

With reference to fig. 1, the present invention mainly includes three processes, namely data acquisition and feature extraction, classifier training and result prediction, which will be described in detail below.

1) Data acquisition and feature extraction

In order to realize auxiliary diagnosis suitable for different age groups according to different eye movement modes of ASD children and normal children in the face emotion recognition process, the influence of local medical level can be limited, and patients with autism spectrum disorder can be accurately evaluated under the condition that professional ability and experience of doctors are insufficient. In the data acquisition process, the eye movement data which are acquired by using the eye movement instrument which is convenient for young children and has low requirement on the professional of doctors are classified by using dynamic daily life scenes as stimulating materials.

The facial stimulation capable of reflecting the specific eye movement pattern of the ASD children is selected around the ubiquitous facial emotion recognition defect of the ASD children aiming at the core problems of face recognition and emotion perception disorder in the ASD children social disorder. Compared with static facial expression stimulation which is commonly used in research, dynamic video emotional stimulation is selected in the data acquisition process. Compared with static picture stimulation, the adopted dynamic video stimulation with daily speech expression is more consistent with daily life scenes, and the emotion recognition capability of the ASD children can be more truly evaluated, because communication occurring in real-world interaction cannot always be well perceived, and meanwhile, the dynamic adjustment of stimulation materials can be carried out according to different age groups of subjects so as to be suitable for children of different age groups.

In one embodiment, the stimulus material in the data collection process consists of 20 film clip videos provided by the Chinese Natural Emotional Audio-Visual Database (CHEAVD) aimed at providing Chinese resources for studying multi-modal and multimedia interactions. For example, six typical and relatively complete emotional stimulus videos were selected for analysis, which were combined from three positive emotional videos and three negative emotional videos. The duration of the video is between 3 seconds and 9 seconds. The data acquisition process used the RED250 eye tracker of SMI, germany, which was integrated into a 22 inch wide screen display panel with a resolution of 1280 x 1024 pixels, to collect eye movement data for each subject in a non-invasive manner. The sampling frequency is 60Hz, and the precision is 0.4 degrees. The freedom of movement of the subject's head is 40 x 20 cm at a distance of 70 cm. During the collection process, on-line eye movement data recording is carried out by using experimental Center of Germany SMI company experimental design software, and off-line data analysis is carried out by using SMI data analysis software BeGaze. Before the experimental tasks are formally performed, the subjects are first trained to understand the tasks in the experiment, and after the understanding tasks, pre-experiments are performed using the samsung tablet ST800 to ensure that the experimenter is familiar with the entire flow of the experiment. The participant was then asked to sit at a distance of about 60-80 cm from the test screen until the eye tracker was able to stably detect the participant's pupil. During the experiment, participants were not able to intervene to avoid any attention bias. Specifically, first, five-point calibration of eye movement is performed: the subject is asked to look at the calibration points in the four corners and middle of the screen in sequence, and the calibration portion is passed only if all five calibration points have errors that do not on average exceed 1 degree viewing angle. During the official experimental task phase, 20 test videos were randomly played and participants allowed to watch each video multiple times because they did not know what was displayed in them.

After the eye movement data is acquired, a Heat Map (Heat Map), a Focus Map (Focus Map) and a Scan Path Map (Scan Path) of each video short film watched by the testee are obtained through the output of the matching data analysis software SMI BeGaze processing. The hotspot graph shows the dynamic change in time and position of the point of regard, for example, in the warm shade of color, i.e., the closer to the color to the right of the color bar in the data analysis software, the longer the time to look at the area. The scan path illustration displays information such as the gaze point position and each gaze time point by point. The focus map displays dynamic changes in the gaze position and time, for example, in brightness.

In conclusion, in task design, a dynamic scene with daily speech expression is adopted as a stimulation material around core problem face recognition and emotion perception obstacle in ASD (ASD-related children) social interaction obstacle, and natural reaction of a subject in social interaction and reality of eye movement data can be reflected better through a real life-activated scene. Furthermore, the eye tracker tracking is non-invasive, and the subject does not need to wear any device when using the eye tracker for data acquisition, so that discomfort is not caused to the subject, especially to children with younger ages. And the stimulation of the experimental task can be flexibly adjusted according to the age condition of the subject, and the diagnosis is also convenient to operate for children of 6-18 months of age.

2) Classification training

After the hot spot graph, the focus graph and the scanning path graph reflecting the eye movement characteristics of the subject are extracted, the neural network classifier for predicting and evaluating the autism is obtained after training and learning by using the neural network.

Referring to fig. 2, the convolutional neural network is adopted in the embodiment of the present invention, and the structure of the whole neural network includes an input layer (input), 3 convolutional layers (i.e., conv1, conv2, conv3), 2 max pooling layers (i.e., max pooling1, max pooling2), 2 full-link layers (i.e., fc1 and fc2), and an output layer (output) using a design similar to the LeNet structure. Specifically, firstly, a data input layer is provided, the input images are respectively a heat map, a focus map and a scanning path map obtained by the eye movement data through analysis software, and the size of the input images is uniformly normalized to 1024 × 1024. The first layer is a convolution layer, the second layer is a maximum pooling layer, the third layer is a convolution layer, the fourth layer is a maximum pooling layer, the fifth layer is a convolution layer, the sixth layer and the seventh layer are all-connected layers, and the eighth layer is an output layer. The activation function of the first six layers is a ReLU nonlinear activation function, the activation function of the seventh layer is a Softmax activation function, the number of neurons of the output layer can be set to be 4, and the four types of activation functions correspond to four categories of healthy, mild autism symptoms, moderate autism symptoms and severe autism symptoms.

After the neural network structure is designed, training of the neural network is initiated for assisted diagnosis of ASD. Specifically, the node numbers of neurons of an input layer, a hidden layer and an output layer and the size of a convolution kernel are set, and weight matrixes are initialized randomly and comprise weight matrixes from the input layer to the hidden layer, from the hidden layer to the hidden layer and from the hidden layer to the output layer. And respectively inputting the heat map, the focus map and the scanning path map of each video short-film eye movement data as the input of the neural network, and training according to a forward propagation algorithm, a backward propagation algorithm and a gradient descent method based on the initialized weight matrix to obtain the weight matrix among all layers of the neural network. And finally, the loss function of the output layer is a cross entropy loss function.

When the effect of the invention is tested, a cross validation method is used, and the effect of the neural network classifier is judged according to a Receiver Operating Characteristic curve (ROC) and an area AUC (area Under ROC) Under the ROC curve by using a classification result confusion matrix. And (3) collecting the classification results of the three classifiers according to the three input images of the eye movement data by adopting a method similar to a Bagging method in ensemble learning so as to give a final classification result.

3) Prediction of classification result

And carrying out classification prediction on the ASD according to the trained neural network classifier model. Firstly, according to different input data, three convolutional neural network classifiers are obtained through training, and the three classifiers respectively take a heat map, a focus map and a scanning path map of eye movement data as input and output four categories corresponding to healthy, mild autism symptoms, moderate autism symptoms and severe autism symptoms. Then, according to the prediction outputs of the three classifiers, a simple voting method is used for combination to give a final prediction result, and the final four results are still one of the four categories.

It should be noted that the neural networks according to the embodiments of the present invention may have the same or different structures, for example, more convolutional layers, fully connected layers, average pooling or maximum pooling may be adopted, and the classification result is not limited to the above four types. Furthermore, the neural network, the result output unit, and the like may be implemented in software or hardware, such as a hardware processor or a logic circuit.

In order to further verify the effect of the invention, the data analysis result of the invention is found to be consistent with the research result through the comparative experimental research of ten ASD children and nineteen normal children. The aoi (area of interest) method is widely used in eye movement analysis. AOI aims to measure the facial regions of interest that the eyes are gazing at, typically including the eyes, nose and mouth, and then count the frequency and time at which the eyes are gazing at these regions. In the pre-experiments of the present invention, ASD children had relatively less fixation time and fixation times in AOI than normal children, more specifically ASD children fixated on the body and objects far beyond the fixation eye, and their thermographic and focal map analysis of eye movement data illustrates the same problem. The invention carries out the auxiliary diagnosis of the ASD based on different eye movement modes.

In conclusion, the invention establishes a technical scheme which is efficient, convenient, easy to popularize and suitable for auxiliary diagnosis of the ASD of young children, aiming at the defects in the conventional ASD auxiliary diagnosis technology and the main problems in ASD screening evaluation and clinical application. Core problems surrounding ASD children social barriers: face recognition and emotion perception disorders aim at eye movement patterns of ASD children different from normal children in face recognition and emotion perception, by means of an infrared vision tracking technology (the eye movement technology can aim at younger and younger ages, and the current eye movement technology can already research infants of 3 months old), and by combining a deep learning method, the complicated and time-consuming manual feature extraction work is omitted, the ASD is diagnosed in an auxiliary mode, early diagnosis and discovery of ASD patients are improved, an effective time window is opened for treatment, early intervention on related patients is realized, the economic burden of society and families is reduced, and accordingly real 'early discovery, early intervention and early treatment' are realized.

It should be noted that, although the steps are described in a specific order, the steps are not necessarily performed in the specific order, and in fact, some of the steps may be performed concurrently or even in a changed order as long as the required functions are achieved.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.

The computer readable storage medium may be a tangible device that retains and stores instructions for use by an instruction execution device. The computer readable storage medium may include, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing.

Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (8)

1. The autism auxiliary assessment system based on deep learning is characterized by comprising a data acquisition and feature extraction unit, a first neural network, a second neural network, a third neural network and a result output unit, wherein the data acquisition and feature extraction unit and the result output unit are respectively in communication connection with the first neural network, the second neural network and the third neural network, and the data acquisition and feature extraction unit and the result output unit are respectively in communication connection with the first neural network, the second neural network and the third neural network, wherein:

the data acquisition and feature extraction unit is used for acquiring eye movement data of a subject watching a dynamic video to obtain a corresponding hotspot graph, a focus graph and a scanning path graph, wherein the hotspot graph is used for representing the time and the dynamic change of the position of a fixation point, the focus graph is used for representing the dynamic change of the fixation position and the time, and the path scanning graph continuously displays the fixation point position and each fixation time information point by point;

the first neural network is used for inputting the heat point diagram to obtain a first classification result;

the second neural network is used for inputting the focus map to obtain a second classification result;

the third neural network is used for inputting the scanning path diagram to obtain a third classification result;

the result output unit collects the first classification result, the second classification result and the third classification result to obtain the autism detection result of the subject.

2. The system of claim 1, wherein the first, second, and third neural networks have the same or different structures.

3. The system of claim 1, wherein the first, second, and third neural networks have the same structure, including an input layer, a first layer of convolutional layers, a second layer of pooling layers, a third layer of convolutional layers, a fourth layer of pooling layers, a fifth layer of convolutional layers, a sixth layer of fully-connected layers, a seventh layer of fully-connected layers, and an output layer.

4. The system of claim 3, wherein the activation functions of the first, second, third, fourth, fifth, and sixth fully-connected layers of the first, second, and third neural networks are ReLU nonlinear activation functions, the activation function of the seventh fully-connected layer is a Softmax activation function, and the number of neurons in the output layer is 4, corresponding to four categories of healthy, mild autistic, moderate autistic, and severe autistic symptoms, respectively.

5. The system of claim 1, wherein the result output unit combines the first classification result, the second classification result, and the third classification result using a simple voting method to give a final predicted result.

6. The system of claim 1, wherein the eye movement data for each subject is collected non-invasively using an eye tracker.

7. The system of claim 1, wherein the heat map displays dynamic changes in time and position of the point of regard in warm color and the focus map displays dynamic changes in time and position of the point of regard in luminance.

8. An autism auxiliary assessment method based on deep learning comprises the following steps:

the method comprises the steps of collecting eye movement data of a subject watching a dynamic video to obtain a corresponding hotspot graph, a focus graph and a scanning path graph, wherein the hotspot graph is used for representing the dynamic change of time and position of a fixation point, the focus graph is used for representing the dynamic change of the fixation position and time, and the path scanning graph continuously displays the position of the fixation point and information of each fixation time point by point;

inputting the distribution of the heat point diagram, the focus diagram and the scanning path diagram into a trained first neural network, a trained second neural network and a trained third neural network to respectively obtain a first classification result, a second classification result and a third classification result;

and aggregating the first classification result, the second classification result and the third classification result to obtain the autism detection result of the subject.

Images