Research & Publications
I study how training, parameterization, data, and feedback shape what AI systems learn and how effectively they perform. A central theme of my work is that seemingly small design choices can have large statistical consequences.
I have shown how attention parameterization changes implicit bias and generalization, how attention heads specialize during training, when learned features can be genuinely recovered, and when synthetic data and verification improve, or fail to improve, a pretrained model. My recent work extends this perspective to agentic systems, where feedback and external computation can substantially change reasoning performance.
My goal is to turn these mechanisms into principles for designing more reliable, interpretable, and efficient AI systems.
Publications
-
Can a One-Layer Transformer Solve Hard Sudoku? Test-Time Compute, Generalization, and Mechanistic Analysis.
Heejune Sheen*, Siyu Chen*, Jianliang He, Awni Altabaa, and Zhuoran Yang. *Equal contribution. Under review. -
Statistical Analysis of Synthetic Data with Verification: A Linear Regression Framework.
Heejune Sheen and Harrison H. Zhou. 2026. Submitted to The Annals of Statistics. [arXiv] -
INFUSER: Influence-Guided Self-Evolution Improves Reasoning.
Siyu Chen, Miao Lu, Beining Wu, Heejune Sheen, Fengzhuo Zhang, Shuangning Li, Zhiyuan Li, Jose Blanchet, Tianhao Wang, and Zhuoran Yang. NeurIPS 2026 (to appear). [arXiv] -
Taming Polysemanticity in LLMs: Theory-Grounded Feature Recovery via Sparse Autoencoders.
Siyu Chen, Heejune Sheen, Xuyuan Xiong, Tianhao Wang, and Zhuoran Yang. ICLR 2026. [arXiv] -
Implicit Regularization of Gradient Flow on One-Layer Softmax Attention.
Heejune Sheen, Siyu Chen, Tianhao Wang, and Harrison H. Zhou. ICLR 2024 Workshop on Bridging the Gap Between Practice and Theory in Deep Learning. [arXiv] Under review. -
Unraveling SARS-CoV-2 Host-Response Heterogeneity through Longitudinal Molecular Subtyping.
Kexin Wang, Yutong Nie, Cole Maguire, Caitlin Syphurs, Heejune Sheen, et al. Preprint, 2024. [bioRxiv] -
Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers.
Siyu Chen*, Heejune Sheen*, Tianhao Wang, and Zhuoran Yang. NeurIPS 2024. *Equal contribution. [arXiv] -
Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality.
Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. COLT 2024; Proceedings of Machine Learning Research 247. [arXiv] -
Tensor Kernel Recovery for Discrete Spatio-Temporal Hawkes Processes.
Heejune Sheen, Xiaonan Zhu, and Yao Xie. IEEE Transactions on Signal Processing, 70, 5859–5870, 2022. -
Exploiting Aggregate Sparsity in Second-Order Cone Relaxations for Quadratic Constrained Quadratic Programming Problems.
Heejune Sheen and Makoto Yamashita. Optimization Methods and Software, 37(2), 753–771, 2022.