Skip to content

🏞️ Official implementation of "Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition"

License

Notifications You must be signed in to change notification settings

louisYen/Gen4Gen

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

56 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

🏞️ Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition

By Chun-Hsiao Yeh*, Ta-Ying Cheng*, He-Yen Hsieh*, Chuan-En Lin, Yi Ma, Andrew Markham, Niki Trigoni, H.T. Kung, Yubei Chen ( * equal contribution)

UC Berkeley, University of Oxford, Harvard University, CMU, HKU, UC Davis

tags: stable diffusion personalized text-to-image generation llm

This repo is the official implementation of "Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition".


TL;DR: We introduce a dataset creation pipeline, Gen4Gen, to compose personal concept into realistic scenes with complex compositions, accompanied by detailed text descriptions.

πŸ“ Updates

  • [Feb 26 2024] [⚑️NEWS⚑️] Added 🏞️ Gen4Gen: dataset creation pipeline code!"
  • [Feb 15 2024] Added demo video πŸ”₯ for "How Gen4Gen Works"

πŸ”Ž Overview of This Repository

  • gen4gen: Toolkits for Gen4Gen pipeline.

🎁 Prepare Personal Assets

Please prepare your personal images and put them under data/s0_source_images. Our personal images are from Unsplash. The structure of πŸ—‚ data/s0_source_images
looks like this:

πŸ—‚ Structure of data/s0_source_images
../data/s0_source_images
└── cat_dog_houseplant_3objs
    β”œβ”€β”€ cat
    β”‚Β Β  └── sergey-semin-agQhOHQipoE-unsplash.jpg
    β”œβ”€β”€ dog
    β”‚Β Β  β”œβ”€β”€ Copy of 5.jpeg
    β”‚Β Β  └── Copy of 6.jpeg
    └── houseplant
        β”œβ”€β”€ Copy of 1.png
        β”œβ”€β”€ Copy of 2.png
        β”œβ”€β”€ Copy of 3.png
        └── Copy of 5.png
└── [folder_of_other_scenes]
    β”œβ”€β”€ [object_name_1]
    β”‚Β Β  β”œβ”€β”€ [image_name_1.jpg]
    β”‚Β Β  β”œβ”€β”€ ...
    β”‚Β Β  └── [image_name_n.jpeg]
    ...
    └── [oject_name_n]

πŸ—ΊοΈ Environments

Library versions
  • pytorch Framework and environment
    • pytorch: 1.13.1
    • cuda: 11.7
    • torchvision: 0.14.1
  • 🧨 Go-to library for diffusion models
    • diffusers: 0.21.4
  • OpenAI OpenAI API for LLM-Guide Object Composition
    • openai: 0.28.1
  • python Programming language
    • python: 3.8.5
  • pytorch Graphics card
    • [Gen4Gen pipeline] GPU: NVIDIA A100-SXM4-40GB x 1 or NVIDIA GeForce RTX 4090-24GB x 1
      • For Step1, Object Association and Foreground Segmentation, requiring around 2.2GB memory footprint
      • For Step3, Background Repainting, requring around 17GB memory footprint

πŸ‘ Acknowledgement

Our codebase is built based on DIS, LLM-grounded Diffusion, SD-XL Inpainting, and custom-diffusion. We really appreciate the authors for the nicely organized code and fantastic works!

πŸ“¬ How to Get Support?

If you have any general questions or need support, please feel free to contact: Chun-Hsiao Yeh, Ta-Ying Cheng and He-Yen Hsieh. Also, we encourage you to open an issue in the GitHub repository. By doing so, you not only receive support but also contribute to the collective knowledge base for others who may have similar inquiries.

❀️ Citation

If you find the codebase and MyCanvas dataset valuable and utilize it in your work, we kindly request that you consider giving our GitHub repository a ⭐ and citing our paper.

@misc{yeh2024gen4gen,
  author        = {Chun-Hsiao Yeh and
                   Ta-Ying Cheng and
                   He-Yen Hsieh and
                   David Chuan-En Lin and
                   Yi Ma and
                   Andrew Markham and
                   Niki Trigoni and
                   H.T. Kung and
                   Yubei Chen},
  title         = {Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition},
  year          = {2024},
  eprint        = {2402.15504},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}

About

🏞️ Official implementation of "Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition"

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages