Foundation models · Multimodal LLMs
Haotian Zhang
Staff Research Scientist · NVIDIA Cosmos Lab
I develop foundation models and multimodal LLMs, with a focus on visual understanding, reasoning, and generation.
At NVIDIA Cosmos Lab, I work with Ming-Yu Liu on foundation models for physical AI. Previously at Apple AI/ML, I worked with Yinfei Yang on multimodal LLMs, including MM1, MM1.5, and the Ferret family. My earlier work at Microsoft Research includes GLIP and GLIPv2.
I received my Ph.D. from the University of Washington, advised by Jenq-Neng Hwang, and my B.S. from Shanghai Jiao Tong University.
Recent updates
- Jun 2026
- Cosmos 3: omnimodal world models for physical AI.
- 2026
- MANZANO at ICLR — unified multimodal understanding and generation.
- Jan 2026
- Joined NVIDIA Cosmos Lab to work on foundation models for physical AI.
Selected research
All publicationsFoundation models, multimodal LLMs, and grounded visual understanding.
-
MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
A hybrid vision tokenizer brings visual understanding and image generation into one scalable multimodal model.
-
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Data and training recipes for multimodal models with stronger text-rich image understanding, grounding, and multi-image reasoning.
-
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Referring, grounding, and reasoning capabilities for understanding mobile user interfaces.
-
GLIPv2: Unifying Localization and Vision-Language Understanding
One model connects object localization with vision-language tasks, from open-vocabulary detection to visual question answering.
Experience
NVIDIA
Staff Research Scientist · Cosmos Lab
2026 – present
Foundation models for physical AI, with Ming-Yu Liu.
Apple
Senior Research Scientist · AI/ML
2023 – 2025
Multimodal LLMs and visual grounding, with Yinfei Yang.
Education
University of Washington
M.S., Applied Mathematics · 2021
M.S., Electrical & Computer Engineering · 2019
Academic service
Vision · Language · Machine learningArea chair
- NeurIPS2025
- Vision & Language
- AAAI2024
- Open-Vocabulary Object Detection
Reviewing & program committees
- ICLR
- CVPR
- ECCV
- ICCV
- NeurIPS
- ICML
- COLM
Also ACL ARR, AAAI, WACV, ICRA, IROS.
Journals IEEE TPAMI · IEEE TCSVT
Workshop organization
- 2024Transformers for Vision Program committee CVPR
- 2023Computer Vision in the Wild Co-organizer CVPR
- 2022Computer Vision in the Wild Co-organizer ECCV
Selected honors
- 2022
Best Paper Finalist
CVPR · GLIP
Grounded Language-Image Pre-training
- 2022
Young Scholar Award
NeurIPS
- 2020–2021
BMTT Challenge team awards
CVPR 2020 · ICCV 2021
Two track wins in 2021; one track win and one runner-up in 2020.
Research & collaboration
Let’s connect.
For research discussions, collaborations, and invited talks on foundation models, multimodal LLMs, and visual understanding.