Multimodal data processing for learning analytics

Learning analytics has been defined as measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimising learning. Multimodal data acquisition and processing techniques have advanced to a stage where it is possible to synchronously process a multitude of information streams about the learning process and the learner, including physiological bodily responses, e.g., brain waves, heart rate, electrodermal activity; responses that can be measured through audio-visual and other sensors, e.g., eye movements, facial expressions, spontaneous body movements; computer interaction data; and in some cases speech data. The joint processing of low-level information reveals significant cues about the learner’s state of mind and emotional state, both of which are effective on the learner’s overall performance and influenced by the instructional settings too.

Multimodal analytics enables the observation and interpretation of interactions and nuances, which would normally be overlooked by traditional methods that rely mainly on manual coding. Introduction of affordable sensors allows access to information from learners’ interactions with the learning (instructional) content in a physical or computer-mediated setting, which could not be possible with traditional log data. The objective is to analyse and model the relationship between automatically observable data and the learning performance leveraging machine learning techniques. Once such a model is built, it can be used to proactively assess the learners’ progress during a learning process and inform the instructor or the learning content (if in an online learning environment) to adapt the instructional setting. Similarly, it can be used to generate developmental feedback for the learner in an intuitive way, which enables the learner to reflect on.

While multi-modal learning analytics and its potential has been examined more widely in the context of STEM subject education, its use in the education of creative subjects (e.g., visual arts, design, digital media, photography etc.) is very limited. The notion of learning outcome is assessed with how well the learner meets certain requirements conforming to appropriate descriptors. But at the same time, further skills are important in the education of creative subjects, such as creativity. To the best knowledge of the researchers, no former research project investigated the power of learning analytics in the context of assessing a meta-skill such as creativity, how the instructional approach influences the exhibited degree of creativity independent from the learning outcomes, and how individuals’ visible cues relate to that.

The project exploits quantitative research methods and a comprehensive experimentation design, which aims to generate a novel dataset. The dataset comprises a diverse set of learners’ responses to online instructional media and a comprehensive recording of multimodal data streams of the learners while engaging with a creative learning task on PC.

This is three-year research project started in October 2019, and is aligned with both the Digital Media Processing, Analytics and AR/VR and Artificial Intelligence, Machine Learning and Data Analytics research themes of the IDT.

For further info, please contact Dr Erhan Ekmekcioglu.