Foundation models for human sensing data
Online
Course provider: Jožef Stefan Institute (JSI)
Instructors: Gašper Slapničar (JSI), Mitja Luštrek (JSI), Amer Mujagić (JSI)
Foundation models are large general-purpose models that can be adapted to various tasks. The most prominent example are in the fields of language (LLMs) and computer vision, but they are increasingly used for other types of sensor data.
The webinar will introduce foundation models in general, as well as the basics of human sensing – biosignals (PPG, ECG, EEG …), motion sensors (accelerometers ...), and tasks for which foundation models are used (blood pressure estimation, stress detection, activity recognition ...). We will dive into modern foundation-model backbones for time series, pretraining paradigms (masked modeling, contrastive learning) and fine-tuning approaches. We will review existing models with a focus on PPG. We will also discuss merging LLMs and sensor-based models that result in hybrid approaches allowing for natural-language interpretation of sensor data.
We are developing a service in SLAIF that will enable fine-tuning of foundation models for human sensing data. Several models and datasets for PPG data are already included, as well as preprocessing and fine-tuning pipelines, and we will give their overview. We will ask for participant feedback on the types of human sensing tasks of interest, which may guide future development to better serve the users.