About Me

I am a second-year Ph.D. student at the University of Maryland, College Park, in the Department of Electrical & Computer Engineering, advised by Prof. Ang Li.
Before that, I obtained my bachelor’s degree at the School of Cyber Science and Engineering, Sichuan University.

I’m always open to collaborations! Feel free to reach out to me at ghsun@umd.edu.

Research Interests

Efficient LLMs
LLM Privacy and Safety Alignment
Pretraining and Reasoning in LLMs

What’s New

[2026.02] Our work What Matters in Transformers? Not All Attention is Needed was accepted at TMLR.
[2025.10] Our work Enhancing the Security of Large Character Set CAPTCHAs Using Transferable Adversarial Examples was accepted at IEEE TDSC.
[2025.05] Started my research internship at Snowflake AI Research, working with Zhewei Yao & Yuxiong He on reinforcement learning for Text-to-SQL.
[2025.05] I am very honored to receive the Qualcomm Innovation Fellowship together with Shwai. Many thanks to Qualcomm for supporting our research on improving the efficiency of the transformer architecture.
[2025.05] Our model Arctic-Text2SQL-R1-32B achieved Top 1 on the BIRD-SQL Leaderboard.
[2024.11] Our work Model-GLUE was accepted at NeurIPS 2024 (Datasets & Benchmarks Track).
[2024.11] Our work SHED and Flora were accepted at NeurIPS 2024.

Publications

ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models

G. Sun, T. Du, K. Feng, C. Luo, X. Ding, Z. Shen, Z. Wang, Y. He, A. Li
arXiv preprint 2026 [Link]

CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs

G. Sun, Z. Wang, B. Tian, M. Liu, Z. Shen, S. He, Y. He, W. Ye, Y. Wang, A. Li
arXiv preprint 2025 [Link]

VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation

Y. Wang*, G. Sun*, W. Ye, G. Qu, A. Li
arXiv preprint 2025 [Link]

Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Services

G. Sun*, Z. Wang*, X. Zhao, B. Tian, Z. Shen, Y. He, J. Xing, A. Li
ResponsibleFM Workshop @ NeurIPS 2025 (Oral) [Link]

Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQL

Z. Yao, G. Sun, L. Borchmann, Z. Shen, M. Deng, B. Zhai, H. Zhang, A. Li, Y. He
arXiv preprint 2025 [Link] [Code]

What Matters in Transformers? Not All Attention is Needed

S. He*, G. Sun*, Z. Shen, A. Li
Transactions on Machine Learning Research (TMLR) 2025 [Link] [OpenReview] [Code]

Enhancing the Security of Large Character Set CAPTCHAs Using Transferable Adversarial Examples

G. Sun*, Y. Fu*, H. Yang, J. Huang, R. Zhang, H. Wang
IEEE Transactions on Dependable and Secure Computing (TDSC) 2025

Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild

X. Zhao*, G. Sun*, R. Cai*, Y. Zhou*, P. Li*, P. Wang, B. Tan, Y. He, L. Chen, Y. Liang, B. Chen, B. Yuan, H. Wang†, A. Li†, Z. Wang†, T. Chen†
NeurIPS 2024 (Datasets & Benchmarks) [Link]

Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers

S. He, T. Ge, G. Sun, et al.
EMNLP 2025 (Main) [Link]

Flora: Federated Fine-tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Z. Wang, Z. Shen, Y. He, G. Sun, et al.
NeurIPS 2024 [Link]

Y. He, Z. Wang, Z. Shen, G. Sun, et al.
NeurIPS 2024 [Link]

Awards

Qualcomm Innovation Fellowship, Qualcomm, 2025
Dean’s Fellowship, University of Maryland, 2024
National Scholarship, Ministry of Education of China, 2022

Guoheng Sun