Skip to content

[CVPR 2024 πŸ”₯] Grounding Large Multimodal Model (GLaMM), the first-of-its-kind model capable of generating natural language responses that are seamlessly integrated with object segmentation masks.

Notifications You must be signed in to change notification settings

mbzuai-oryx/groundingLMM

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

42 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

GLaMM : Pixel Grounding Large Multimodal Model [CVPR 2024]

Oryx Video-ChatGPT

Mohamed bin Zayed University of AI, Australian National University, Aalto University, Carnegie Mellon University, University of California - Merced, LinkΓΆping University, Google Research

paper Dataset Demo Website video


πŸ“’ Latest Updates

  • Mar-21-24- We're excited to announce the release of GranD dataset and the GranD Automated Annotation Pipeline πŸ”₯
  • Feb-27-23- We're thrilled to share that GLaMM has been accepted to CVPR 2024! 🎊
  • Dec-27-23- GLaMM training and evaluation codes, pretrained checkpoints and GranD-f dataset are released click for details πŸ”₯πŸ”₯
  • Nov-29-23: GLaMM online interactive demo is released demo link. πŸ”₯
  • Nov-07-23: GLaMM paper is released arxiv link. 🌟
  • 🌟 Featured: GLaMM is now highlighted at the top on AK's Daily Papers page on HuggingFace! 🌟

GLaMM Overview

Grounding Large Multimodal Model (GLaMM) is an end-to-end trained LMM which provides visual grounding capabilities with the flexibility to process both image and region inputs. This enables the new unified task of Grounded Conversation Generation that combines phrase grounding, referring expression segmentation, and vision-language conversations. Equipped with the capability for detailed region understanding, pixel-level groundings, and conversational abilities, GLaMM offers a versatile capability to interact with visual inputs provided by the user at multiple granularity levels.


πŸ† Contributions

  • GLaMM Introduction. We present the Grounding Large Multimodal Model (GLaMM), the first-of-its-kind model capable of generating natural language responses that are seamlessly integrated with object segmentation masks.

  • Novel Task & Evaluation. We propose a new task of Grounded Conversation Generation (GCG). We also introduce a comprehensive evaluation protocol for this task.

  • GranD Dataset Creation. We create the GranD - Grounding-anything Dataset, a large-scale densely annotated dataset with 7.5M unique concepts grounded in 810M regions.


πŸš€ Dive Deeper: Inside GLaMM's Training and Evaluation

Delve into the core of GLaMM with our detailed guides on the model's Training and Evaluation methodologies.

  • Installation: Provides guide to set up conda environment for running GLaMM training, evaluation and demo.

  • Datasets: Provides detailed instructions to download and arrange datasets required for training and evaluation.

  • GranD: Provides detailed instructions to download the GranD dataset and run the automated annotation pipeline.

  • Model Zoo: Provides downloadable links to all pretrained GLaMM checkpoints.

  • Training: Provides instructions on how to train the GLaMM model for its various capabilities including Grounded Conversation Generation (GCG), Region-level captioning, and Referring Expression Segmentation.

  • Evaluation: Outlines the procedures for evaluating the GLaMM model using pretrained checkpoints, covering Grounded Conversation Generation (GCG), Region-level captioning, and Referring Expression Segmentation, as reported in our paper.

  • Demo: Guides you through setting up a local demo to showcase GLaMM's functionalities.

πŸ‘οΈπŸ’¬ GLaMM: Grounding Large Multimodal Model

The components of GLaMM are cohesively designed to handle both textual and optional visual prompts (image level and region of interest), allowing for interaction at multiple levels of granularity, and generating grounded text responses.

GLaMM Architectural Overview


πŸ” Grounding-anything Dataset (GranD)

The Grounding-anything GranD dataset, a large-scale dataset with automated annotation pipeline for detailed region-level understanding and segmentation masks. GranD comprises 7.5M unique concepts anchored in a total of 810M regions, each with a segmentation mask.

Dataset Annotation Pipeline


Below we present some examples of the GranD dataset.

GranD Dataset Sample

GranD Dataset Sample


πŸ“š Building GranD-f for Grounded Conversation Generation

The GranD-f dataset is designed for the GCG task, with about 214K image-grounded text pairs for higher-quality data in fine-tuning stage.

GranD-f Dataset Sample


πŸ€– Grounded Conversation Generation (GCG)

Introducing GCG, a task to create image-level captions tied to segmentation masks, enhancing the model’s visual grounding in natural language captioning.

Results_GCG

GCG_Table


πŸš€ Downstream Applications

🎯 Referring Expression Segmentation

Our model excels in creating segmentation masks from text-based referring expressions.

Results_RefSeg

Table_RefSeg


πŸ–ΌοΈ Region-Level Captioning

GLaMM generates detailed region-specific captions and answers reasoning-based visual questions.

Results_RegionCap

Table_RegionCap


πŸ“· Image Captioning

Comparing favorably to specialized models, GLaMM provides high-quality image captioning.

Results_Cap


πŸ’¬ Conversational Style Question Answering

GLaMM demonstrates its prowess in engaging in detailed, region-specific, and grounded conversations. This effectively highlights its adaptability in intricate visual-language interactions and robustly retaining reasoning capabilities inherent to LLMs.

Results_Conv


Results_Conv


πŸ“œ Citation

  @article{hanoona2023GLaMM,
          title={GLaMM: Pixel Grounding Large Multimodal Model},
          author={Rasheed, Hanoona and Maaz, Muhammad and Shaji, Sahal and Shaker, Abdelrahman and Khan, Salman and Cholakkal, Hisham and Anwer, Rao M. and Xing, Eric and Yang, Ming-Hsuan and Khan, Fahad S.},
          journal={The IEEE/CVF Conference on Computer Vision and Pattern Recognition},
          year={2024}
  }

πŸ™ Acknowledgement

We are thankful to LLaVA, GPT4ROI, and LISA for releasing their models and code as open-source contributions.


About

[CVPR 2024 πŸ”₯] Grounding Large Multimodal Model (GLaMM), the first-of-its-kind model capable of generating natural language responses that are seamlessly integrated with object segmentation masks.

Topics

Resources

Stars

Watchers

Forks