CASPER
Cross-Modal Alignment for Semantic Processing and Effective Representation

Research goals

• Learn shared semantic spaces across text, image, audio/speech, and video with explicit cross-modal alignment objectives
• Improve grounding and robustness of multimodal generation and reasoning via graph-structured representations
• Enable neural decoding of speech/audio (and related percepts) from EEG/MEG/fMRI with strong generalization and uncertainty awareness
• Release reproducible artifacts (code, models, evaluation protocols) aligned with open science practices

Methodology

• Graph-based multimodal fusion: hierarchical aggregation and concept-level representations using GNNs
• Alignment + supervised fine-tuning: cross-modal contrastive/objective design + task-specific adaptation on curated multimodal datasets
• Neural foundation modeling: unified encoders for EEG/MEG/fMRI with cross-subject/session normalization and efficient decoding heads
• Evaluation & reliability: benchmarking for factual/semantic grounding, robustness across domains, and neural reconstruction fidelity

Multi-modal representation learning

SO1
Develop a human-like multimodal agent that learns unified, graph-structured representations across text, image, audio/speech, and video for robust semantic understanding and generation.

Key activities

• Build concept-level graphs from grid/patch and token features
• Learn aligned multimodal embeddings with contrastive and structured objectives
• Evaluate grounding, compositionality, and robustness across modalities
• Release reusable representation and evaluation components

Supervised fine-tuning and optimization

SO2
Refine and optimize the multimodal agent through supervised fine-tuning on curated multimodal datasets, improving reliability, generalization, and controllability.

Key activities

• Curate training mixtures and task formulations across modalities
• Fine-tune with quality/robustness targets and safety-aware evaluation
• Improve efficiency via parameter-efficient adaptation and scalable training
• Establish benchmarking protocols for generalization and failure modes

Neural representation learning and decoding (EEG/MEG/fMRI)

SO3
Develop unified neural encoders and decoding pipelines that map brain activity to aligned multimodal semantics, enabling neural-to-speech/audio reconstruction with strong cross-subject transfer.

Key activities

• Build modality-agnostic encoders for EEG/MEG/fMRI time series
• Align neural embeddings with multimodal semantic spaces (text/audio/image)
• Develop efficient decoders for speech/audio reconstruction and evaluation
• Validate generalization across subjects, sessions, and datasets