Multimodal Hybrid Deep Learning for the Detection of Autism Spectrum Disorder (ASD) Using Eye-Tracking Data
DOI:
https://doi.org/10.59261/jequi.v8i5.441Keywords:
Autism Spectrum Disorder, Deep Learning, Eye Tracking, Late Fusion, Multimodal LearningAbstract
Background: Autism spectrum disorder (ASD) assessment may benefit from objective measures of visual attention, although eye-tracking classifiers remain research tools rather than replacements for clinical diagnosis.
Objective: This study compared eye-tracking representations and sequence encoders for binary classification of ASD versus typical development using the Saliency4ASD dataset.
Methods: Eye-movement sequences, fixation maps, stimulus images, and tabular features combining gaze and stimulus content were investigated in a hybrid learning pipeline that combined sequence encoders, ResNet-18 image-processing branches, and a multilayer perceptron (MLP) through feature concatenation. The sequential experiments evaluated six data-partition ratios, 15 modality combinations, seven optimization configurations, and alternative sequence architectures. The selected model was also evaluated across five additional holdout partitions grouped by stimulus image.
Results: The combination of scanpath and fixation-map representations achieved a file-level F1 score of 0.866 during modality confirmation. The long short-term memory (LSTM) + gated recurrent unit (GRU) configuration achieved a file-level F1 score of 0.9333 and a receiver operating characteristic area under the curve (ROC-AUC) of 0.9733 on the originally selected test partition comprising 30 stimulus images. Five additional image-grouped partitions yielded a mean file-level F1 score of 0.839 ± 0.015. The random forest baseline achieved an F1 score of 0.8999 on the original partition; however, it was not evaluated across the additional partitions.
Conclusion: The results demonstrate the potential of combining temporal and spatial gaze representations while highlighting sensitivity to data-partition selection. Reuse of test data during model tuning, limited subject independence, and the absence of verified generalization to previously unseen children preclude claims of independently validated diagnostic performance.
Downloads
References
Alzubaidi, L., Al-Amidie, M., Al-Asadi, A., Humaidi, A. J., Al-Shamma, O., Fadhel, M. A., Zhang, J., Santamaría, J., & Duan, Y. (2021). Novel transfer learning approach for medical imaging with limited labeled data. Cancers, 13(7), 1590. https://doi.org/10.3390/cancers13071590
Chen, H., Zhang, X., Chen, Z., Ren, Y., & Liu, R. (2025). Auxiliary diagnostic method for children with autism spectrum disorder based on virtual reality and eye-tracking technology. Scientific Reports, 15(1), 40552. https://doi.org/10.1038/s41598-025-24243-w
Chen, J., Liao, M., Wang, G., & Chen, C. (2020). An intelligent multimodal framework for identifying children with autism spectrum disorder. International Journal of Applied Mathematics and Computer Science, 30(3), 435–448. https://doi.org/10.34768/amcs-2020-0032
Comparan‐Meza, M., Vargas de la Cruz, I., Jauregui‐Huerta, F., Gonzalez‐Castañeda, R. E., Gonzalez‐Perez, O., & Galvez‐Contreras, A. Y. (2021). Biopsychological correlates of repetitive and restricted behaviors in autism spectrum disorders. Brain and behavior, 11(10), e2341. https://doi.org/10.1002/brb3.2341
de Belen, R. A. J., Eapen, V., Bednarz, T., & Sowmya, A. (2024). Using visual attention estimation on videos for automated prediction of autism spectrum disorder and symptom severity in preschool children. PLoS ONE, 19(2), e0282818. https://doi.org/10.1371/journal.pone.0282818
Duan, H., Zhai, G., Min, X., Zhu, Y., Fang, Y., & Yang, X. (2019). A dataset of eye movements for the children with autism spectrum disorder. Dalam Proceedings of the 10th ACM Multimedia Systems Conference (MMSys’19) (255–260). https://doi.org/10.1145/3304109.3325818
Fan, Y., Tang, Q., Guo, Y., & Wei, Y. (2024). BiLSTM-MLAM: A multi-scale time series prediction model for sensor data based on Bi-LSTM and local attention mechanisms. Sensors, 24(12), 3962. https://doi.org/10.3390/s24123962
Han, J., Jiang, G., Ouyang, G., & Li, X. (2022). A multimodal approach for identifying autism spectrum disorders in children. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 30, 2003–2016. https://doi.org/10.1109/TNSRE.2022.3192431
Kanhirakadavath, M. R., & Chandran, M. S. M. (2022). Investigation of eye-tracking scan path as a biomarker for autism screening using machine learning algorithms. Diagnostics, 12(2), 518. https://doi.org/10.3390/diagnostics12020518
Mazumdar, P., Arru, G., & Battisti, F. (2021). Early detection of children with autism spectrum disorder based on visual exploration of images. Signal Processing: Image Communication, 94, 116184. https://doi.org/10.1016/j.image.2021.116184
Mumenin, N., Yousuf, M. A., Nashiry, M. A., Azad, A. K. M., Alyami, S. A., Liò, P., & Moni, M. A. (2024). ASDNet: A robust involution-based architecture for diagnosis of autism spectrum disorder using eye-tracking. IET Computer Vision, 18(5), 666–681. https://doi.org/10.1049/cvi2.12271
Nurhasan, A. A., & Kasprowski, P. (2026). Application of eye movement analysis in medicine: A review across neurodevelopmental, neurological, and neurodegenerative disorders. Applied Sciences, 16(5), 2548. https://doi.org/10.3390/app16052548
Parikh, J., Goyal, S., Malhotra, Y., & Leong, W. Y. (2026). A spatiotemporal hybrid neural network for robust autism spectrum disorder detection via eye tracking feature fusion. Discover Artificial Intelligence, 6, 495. https://doi.org/10.1007/s44163-026-01458-y
Rakotomanana, H., & Rouhafzay, G. (2025). A scoping review of AI-based approaches for detecting autism traits using voice and behavioral data. Bioengineering, 12(11), 1136. https://doi.org/10.3390/bioengineering12111136
Salari, N., Rasoulpoor, S., Rasoulpoor, S., Shohaimi, S., Jafarpour, S., Abdoli, N., Khaledi-Paveh, B., & Mohammadi, M. (2022). The global prevalence of autism spectrum disorder: A comprehensive systematic review and meta-analysis. Italian Journal of Pediatrics, 48, 112. https://doi.org/10.1186/s13052-022-01310-w
Simon, J., Kapileswar, N., Datchinamoorthi, M., Muthukumar, S., & Keerthana Devi, G. (2024). Gaze-assisted autism spectrum disorder identification: A fusion of machine learning and deep learning approaches for preemptive identification. Dalam Proceedings of the 5th International Conference on Mobile Computing and Sustainable Informatics (ICMCSI) (525–529). https://doi.org/10.1109/ICMCSI61536.2024.00082
Startsev, M., & Dorr, M. (2019). Classifying autism spectrum disorder based on scanpaths and saliency. Dalam Proceedings of the IEEE International Conference on Multimedia & Expo Workshops (ICMEW) (633–636). https://doi.org/10.1109/ICMEW.2019.00122
Thanarajan, T., Alotaibi, Y., Rajendran, S., & Nagappan, K. (2023). Eye-tracking based autism spectrum disorder diagnosis using chaotic butterfly optimization with deep learning model. Computers, Materials & Continua, 76(2), 1995–2013. https://doi.org/10.32604/cmc.2023.039644
Tsai, C. H., Chen, K. L., Li, H. J., Chen, K. H., Hsu, C. W., Lu, C. H., ... & Huang, C. Y. (2020). The symptoms of autism including social communication deficits and repetitive and restricted behaviors are associated with different emotional and behavioral problems. Scientific reports, 10(1), 20509. https://doi.org/10.1038/s41598-020-76292-y
Wang, L., Wang, B., Wu, C., Wang, J., & Sun, M. (2023). Autism spectrum disorder: neurodevelopmental risk factors, biological mechanism, and precision therapy. International journal of molecular sciences, 24(3), 1819. https://doi.org/10.3390/ijms24031819
White, J., & Power, S. D. (2023). k-fold cross-validation can significantly over-estimate true classification accuracy in common EEG-based passive BCI experimental designs: An empirical investigation. Sensors, 23(13), 6077. https://doi.org/10.3390/s23136077
Zhu, F.-L., Wang, S.-H., Liu, W.-B., Zhu, H.-L., Li, M., & Zou, X.-B. (2023). A multimodal machine learning system in early screening for toddlers with autism spectrum disorders based on the response to name. Frontiers in Psychiatry, 14, 1039293. https://doi.org/10.3389/fpsyt.2023.1039293
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Citra Wanodya Prameswari, Erwin Budi Setiawan
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA). that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
