Ishan Misra

Research Scientist at GenAI
[Google Scholar] [Github] [CV]

My research interest is in multimodal, generative and self-supervised computer vision. I am a Research Scientist at GenAI Research (Meta) where I lead the generative efforts on video. I was previously part of the Fundamental AI Research (FAIR) team at Meta working on self-supervised learning. I finished my PhD at the Robotics Institute at Carnegie Mellon University where I worked with Martial Hebert and Abhinav Gupta. My PhD Thesis was titled “Visual Learning with Minimal Human Supervision” for which I received the SCS Distinguished Dissertation Award (Runner Up) 2018. For my work in self-supervised learning, I was featured in the MIT Tech Review’s 35 innovators under 35 list (compiled globally across technological disciplines). You can hear me on Lex Fridman’s podcast for a fun overview of my work.


Collaborators and Interns

  • Saketh Rambhatla Postdoctoral researcher at Meta. PhD from University of Maryland, College Park
  • Xudong Wang (University of California, Berkeley). Hosted at FAIR with Rohit Girdhar.
  • Yue Zhao (University of Texas, Austin). Hosted at FAIR with Rohit Girdhar.
  • Xingyi Zhou (University of Texas, Austin). Hosted at FAIR with Rohit Girdhar and Armand Joulin.
  • Bowen Cheng (University of Illinois, Urbana Champaign). Hosted at FAIR with Rohit Girdhar and Alex Kirillov.
  • Karan Desai (University of Michigan, Ann Arbor). Hosted at FAIR with Laurens van der Maaten.
  • Zaiwei Zhang (University of Texas, Austin). Hosted at FAIR with Rohit Girdhar and Armand Joulin.
  • Zhongzheng (Jason) Ren (University of Illinois, Urbana Champaign). Hosted at FAIR with Rohit Girdhar.
  • Yuki Asano (University of Oxford). Hosted at FAIR with Armand Joulin, Piotr Bojanowski, and Andrea Vedaldi.
  • Pedro Morgado (University of California, San Diego).
  • Huaizu Jiang (University of Massachusetts, Amherst). Hosted at FAIR with Xinlei Chen and Marcus Rohrbach.
  • Jyh-Jing Hwang (University of California, Berkeley). Hosted at FAIR with Laurens van der Maaten.
  • Yan Wang (Cornell University). Hosted at FAIR with Laurens van der Maaten.
  • Terrance de Vries (University of Guelph). Hosted at FAIR with Laurens van der Maaten.


Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
Rohit Girdhar**, Mannat Singh**, Andrew Brown*, Quentin Duval*, Samaneh Azadi*, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, Ishan Misra*
InstanceDiffusion: Instance-level Control for Image Generation
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, Ishan Misra
CVPR 2024
Generating Illustrated Instructions
Sachit Menon, Ishan Misra, Rohit Girdhar
CVPR 2024
FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis
Feng Liang, Bichen Wuand Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, Diana Marculescu
CVPR 2024
VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation
Xudong Wang, Ishan Misra, Ziyun Zheng, Rohit Girdhar, Trevor Darrell
CVPR 2024
The effectiveness of MAE pre-pretraining for billion-scale pretraining
Mannat Singh*, Quentin Duval*, Kalyan Vasudev Alwala*, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, Rohit Girdhar, Ishan Misra
ICCV 2023
MOST: Multiple Object localization with Self-supervised Transformers for object discovery.
Saketh Rambhatla, Ishan Misra, Rama Chellappa, Abhinav Shrivastava
ICCV 2023 (Oral)
MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Poses
Yang Fu, Ishan Misra, Xiaolong Wang
ICML 2023
ImageBind: One Embedding Space To Bind Them All
Rohit Girdhar*, Alaaeldin El-Nouby*, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra*
CVPR 2023 (Highlighted paper)
Cut and Learn for Unsupervised Object Detection and Instance Segmentation
Xudong Wang, Rohit Girdhar, Stella X. Yu, Ishan Misra
CVPR 2023
Learning Video Representations from Large Language Models
Yue Zhao, Ishan Misra, Philipp Krahenbuhl, Rohit Girdhar
CVPR 2023 (Highlighted paper)
The Hidden Uniform Cluster Prior in Self-Supervised Learning
Mahmoud Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Nicolas Ballas
ICLR 2023
OmniMAE: Single Model Masked Pretraining on Images and Videos
Rohit Girdhar*, Alaaeldin El-Nouby*, Mannat Singh*, Kalyan Vasudev Alwala*, Armand Joulin, Ishan Misra*
CVPR 2023
Masked Siamese Networks for Label-Efficient Learning
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas
ECCV 2022
Detecting Twenty-thousand Classes using Image-level Supervision
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krahenbuhl, Ishan Misra
ECCV 2022
Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, Piotr Bojanowski
Omnivore: A Single Model for Many Visual Modalities
Rohit Girdhar*, Mannat Singh*, Nikhila Ravi*, Laurens van der Maaten, Armand Joulin, Ishan Misra*
CVPR 2022 (Oral)
Masked-attention Mask Transformer for Universal Image Segmentation
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar
CVPR 2022
An End-to-End Transformer Model for 3D Object Detection
Ishan Misra, Rohit Girdhar, Armand Joulin
ICCV 2021 (Oral)
Emerging Properties in Self-Supervised Vision Transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin
ICCV 2021
Self-Supervised Pretraining of 3D Features on any Point-Cloud
Zaiwei Zhang, Rohit Girdhar, Armand Joulin, Ishan Misra
ICCV 2021
MDETR : Modulated Detection for End-to-End Multi-Modal Understanding
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, Nicolas Carion
ICCV 2021 (Oral)
Audio-Visual Instance Discrimination with Cross-Modal Agreement
Pedro Morgado, Nuno Vasconcelos, Ishan Misra
CVPR 2021 (Best Paper Candidate)
Robust Audio-Visual Instance Discrimination
Pedro Morgado, Ishan Misra, Nuno Vasconcelos
CVPR 2021 (Oral)
Barlow Twins: Self-Supervised Learning via Redundancy Reduction
Jure Zbontar*, Li Jing*, Ishan Misra, Yann LeCun, Stéphane Deny
ICML 2021
3D Spatial Recognition without Spatially Labeled 3D
Zhongzheng Ren, Ishan Misra, Alexander G. Schwing, Rohit Girdhar
CVPR 2021
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin
NeurIPS 2020
Self-Supervised Learning of Pretext-Invariant Representations
Ishan Misra, Laurens van der Maaten
CVPR 2020
ClusterFit: Improving Generalization of Visual Representations
Xueting Yan*, Ishan Misra*, Abhinav Gupta, Deepti Ghadiyaram**, Dhruv Mahajan**
CVPR 2020
In Defense of Grid Features for Visual Question Answering
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, Xinlei Chen
CVPR 2020
3D-RelNet: Joint Object and Relational Network for 3D Prediction
Nilesh Kulkarni, Ishan Misra, Shubham Tulsiani, Abhinav Gupta
ICCV 2019
Scaling and Benchmarking Self-Supervised Visual Representation Learning
Priya Goyal, Dhruv Mahajan, Abhinav Gupta*, Ishan Misra*
ICCV 2019
Binary Image Selection (BISON): Interpretable Evaluation of Visual Grounding
Hexiang Hu, Ishan Misra, Laurens van der Maaten
ICCV Workshop on Vision and Language, 2019
Does Object Recognition Work for Everyone?
Terrance DeVries*, Ishan Misra*, Changhan Wang*, Laurens van der Maaten
CVPR 2019 Workshop on Computer Vision for Global Challenges (CV4GC)
Mainstream: Dynamic Stem-Sharing for Multi-Tenant Video Processing
Angela Jiang, Daniel L.-K. Wong, Christopher Canel, Ishan Misra, Michael Kaminsky, Michael Kozuch, Padmanabhan Pillai, David G. Andersen and Gregory Ganger
USENIX Annual Technical Conference 2018
Learning by Asking Questions
Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta and Laurens van der Maaten
CVPR 2018 (Oral)
Cut Paste and Learn: Surprisingly Easy Synthesis
for Instance Detection
Debidatta Dwibedi, Ishan Misra and Martial Hebert
ICCV 2017
From Red Wine to Red Tomato: Composition with Context
Ishan Misra, Abhinav Gupta and Martial Hebert
CVPR 2017 (Oral) [Acceptance Rate 2.65%]
Shuffle and Learn: Unsupervised Learning
using Temporal Order Verification
Ishan Misra, C. Lawrence Zitnick and Martial Hebert
ECCV 2016
Seeing through the Human Reporting Bias:
Visual Classifiers from Noisy Human-Centric Labels
Ishan Misra, C. Lawrence Zitnick, Margaret Mitchell and Ross Girshick
CVPR 2016
Cross-stitch Networks for Multi-Task Learning
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta and Martial Hebert
CVPR 2016 (Spotlight) [Acceptance Rate 9.7%]
Generating Natural Questions About an Image
Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin,
Margaret Mitchell, Xiaodong He and Lucy Vanderwende
ACL 2016 (Oral) (Long Paper)
Visual Storytelling
Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aishwarya Agrawal,
Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra,
C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley and Margaret Mitchell.
NAACL 2016
Watch and Learn: Semi-Supervised Learning of Object Detectors from Video
Ishan Misra, Abhinav Shrivastava and Martial Hebert
CVPR 2015
Applying artificial vision models to human scene understanding
E. Aminoff, M. Toneva, A. Shrivastava, X. Chen, I. Misra, A. Gupta, M. Tarr.
Journal of Frontiers in Computational Neuroscience, 2015
Data-driven Exemplar Model Selection
Ishan Misra, Abhinav Shrivastava and Martial Hebert
WACV 2014 (Oral) (Best Student Paper)
Hybrid Implementation of Error Diffusion Dithering
Aditya Deshpande, Ishan Misra and P J Narayanan
High Performance Computing (HiPC) 2011


Optimizing multi-class multimedia data classification using negative data
Xian-Sheng Hua, Jin Li and Ishan Misra
Optimizing multi-class image classification using patch features
Ishan Misra, Jin Li and Xian-Sheng Hua