PAIR Diffusion: A Comprehensive Multimodal
Object-level Image Editor

1Picsat AI Research, 2UTrento, 3UT Austin 4UC Berkley 5UIUC 6Georgia Tech
PAIR diffusion achieves comprehensive editing capabilities in a multimodal manner by percieving images as composition of various objects.

Abstract

Generative image editing has recently witnessed extremely fast-paced growth. Some works use high-level conditioning such as text, while others use low-level conditioning. Nevertheless, most of them lack fine-grained control over the properties of the different objects present in the image, i.e.\,object-level image editing. In this work, we tackle the task by perceiving the images as an amalgamation of various objects and aim to control the properties of each object in a fine-grained manner. Out of these properties, we identify structure and appearance as the most intuitive to understand and useful for editing purposes. We propose PAIR Diffusion, a generic framework that can enable a diffusion model to control the structure and appearance properties of each object in the image. We show that having control over the properties of each object in an image leads to comprehensive editing capabilities. Our framework allows for various object-level editing operations on real images such as reference image-based appearance editing, free-form shape editing, adding objects, and variations. Thanks to our design, we do not require any inversion step. Additionally, we propose multimodal classifier-free guidance which enables editing images using both reference images and text when using our approach with foundational diffusion models. We validate the above claims by extensively evaluating our framework on both unconditional and foundational diffusion models.

Inference Overview

  • Appearance Editing
  • Add Objects and Shape editing
  • Multimodal Control

Results

Appearance Editing

Original Image

Reference Image

Edited Image

Original Image

Reference Image

Edited Image

Original Image

Reference Image

Prompt - A picture of sky with birds in it

Edited Image

Add Objects and Shape Editing

Original Image

Reference Image

Prompt - A picture of sandwich on table

Edited Image

Original Image

Reference Image

Prompt - A picture of the cake on the table

Edited Image

Original Image

Reference Image

Prompt - A picture of the crystal clear lake with reflections

Edited Image

Object Variations

Original Image

Edited Image

Original Image

Edited Image

Prompt - A picture of a purple door.

Original Image

Edited Image

BibTeX

@inproceedings{goel2024pair,
            title={PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor},
            author={Goel, Vidit and Peruzzo, Elia and Jiang, Yifan and Xu, Dejia and Xu, Xingqian and Sebe, Nicu and Darrell, Trevor and Wang, Zhangyang and Shi, Humphrey},
            booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
            pages={8609--8618},
            year={2024}}
@article{goel2023pair,
        title={PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models},
        author={Goel, Vidit and Peruzzo, Elia and Jiang, Yifan and Xu, Dejia and Sebe, Nicu and Darrell, Trevor and 
        Wang, Zhangyang and Shi, Humphrey},
        journal={arXiv preprint arXiv:2303.17546},
        year={2023}}