Object-centric deep generative models

Deep generative models are capable of generating very realistic images. However, such models usually do not allow control on the level of individual objects. In this project we aim to overcome to shortcoming. In particular, slot attention is an effective mathematical approach for object-centric representations
in computer vision tasks without requiring any supervision. Despite its
object-centric binding ability brought by compositional modelling, as a deterministic
module, slot attention lacks the ability to generate novel scenes. In this paper,
we propose the Slot-VAE, a generative model that integrates slot attention with the
hierarchical VAE framework for object-centric structured image generation. From
each image, the model simultaneously infers a global scene representation to capture
high-level scene structure and object-centric slot representations to embed
individual object components. During generation, slot representations are generated
from global scene representation to ensure coherent scene structure. Our
experiments demonstrate that Slot-VAE achieves better sample quality and scene
structure accuracy compared to slot representation-based baselines.

In this project, we will extend our Slot-VAE models in various directions. One avenue is to create more compact models with similar capabilities that can be trained faster and with a smaller memory footprint. Another potential avenue is to consider multiple layers in the hierarchical model, potentially allowing such models to capture more complex relationships between objects. Moreover, the student may also explore applying such models to signals (e.g., electroencephalograms) instead of images, leading to novel procedures to generate complex synthetic signals and potentially new approaches to signal interpretation.

Requirements

Passion for methodological aspects of machine learning models. Good knowledge of probability and information theory.

Contact

Last modified: 2023-12-16