Multi-View Digital Representation Of Social Behaviours In Children And Action Recognition Methods

Abstract

Autism spectrum disorders (ASD) affect at least 1% of children globally. It is partially defined by social behaviour delays in eye contact and joint attention during social interaction and with evidence of reduced heart rate variability (HRV) under static and social stress environments. Currently, no validated artificial intelligence or signal processing algorithms are available to objectively quantify behavioural and physiological markers in unrestricted interactive play environments to assist in the diagnosis of ASD. This thesis proposes that social behavioural and physiological markers of children with ASD can be objectively quantified through a synergistic digital approach from multi-modal and multi-view data sources. First, a novel deep learning (DL) framework for social behaviour recognition using a fusion of multi-view and multi-modal predictions is proposed. It utilises true-colour images and moving trajectory (optical flow) images extracted from fixed camera video recordings to detect eye contact between children and caregivers in free play while elucidating unique digital features of eye contact behaviour in multiple individual social interaction settings. Moreover, for the first time, a support vector machine model with feature selection is implemented along with statistical analysis, to identify effective facial features and facial orientations for use in identifying ASD during joint attention episodes in free play. Furthermore, a customised NeuroKit2 toolbox was validated using the opensource QT database and a clinical baseline social interaction task. This toolbox facilitates the automated extraction of HRV metrics and allows between-group comparisons in physiological markers. The work highlights the importance of developing explainable algorithms that objectively quantifying multi-modal digital markers. It offers the potential for the use of digitalised phenotypes to aid in the assessment of ASD and intervention in naturalistic social interaction.