Emergence and Effectiveness of Task Vectors in In-Context Learning
Encoder-decoder analysis of compact task vectors that drive in-context predictions.
Overview figure for the task-vector analysis in in-context learning.
Summary
TL;DR: We studied how transformers form compact task vectors during in-context learning and how those vectors drive predictions.
Why does in-context learning (ICL) succeed or fail depending on the task? We explored this through an encoder-decoder perspective: how a model encodes the task from context examples and decodes that task information into predictions.
Task-Vector Formation
The paper analyzes task vectors as compact internal representations that emerge during pretraining and support in-context predictions.
Key Findings
The main finding is that transformer models can form compact task vectors that causally contribute to in-context predictions.
Causal Analysis
We performed intervention analyses supporting the claim that task-vector representations are not only correlated with ICL behavior, but help drive it.
- Representation analysis: Track compact task information in intermediate representations
- Causal interventions: Test whether modifying task-vector directions changes predictions
- Encoder-decoder framing: Separate task encoding from prediction decoding
Citation
@article{song2024context,
title={Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective},
author={Song, Jinyeop and Han, Seungwook and Agrawal, Pulkit and Gore, Jeffrey},
journal={ICML 2025 (Spotlight)},
year={2024}
}
Paper: arXiv:2412.12276 - ICML 2025 Spotlight
