Remembering the Unseen: How PERSISTBENCH Challenges 4D Foundation Models to Enhance Visual Memory

Recent research at Cornell University has revealed a significant gap in how today's 4D foundation models remember the visual world. While these models excel in reconstructing dynamic environments, they lack the ability to recall objects once they leave the camera's view. This new insight is encapsulated in a benchmark called PERSISTBENCH, which methodically assesses visual memory and could reshape the development of future AI systems.

What is PERSISTBENCH?

PERSISTBENCH is an innovative dataset and metric suite created to evaluate the memory capabilities of 4D foundation models. Unlike traditional metrics that focus on pixel accuracy, PERSISTBENCH introduces three evaluation criteria: object permanence, motion continuity, and appearance preservation. This approach allows researchers to measure how well AI systems can maintain an understanding of dynamic scenes, even when parts of those scenes are no longer visible.

The Investigation into Memory

The research team conducted extensive evaluations using varied 4D models to uncover how these systems perform when objects move out of frame. The findings highlighted a troubling trend: most models exhibited a significant drop in performance for scenes where objects were not visible, underscoring the fact that "seeing is not remembering." This is mainly attributed to the models' training on datasets where objects remain continuously visible, leaving them ill-equipped to handle situations where objects disappear.

Key Metrics of Visual Memory

To fill the assessment gap, PERSISTBENCH utilizes 360-degree videos as an omniscient ground truth. The three primary metrics introduced are:

• Object Permanence: Evaluates if the model understands that an object continues to exist even when it leaves the frame.
• Motion Continuity: Assesses whether the model can predict the motion trajectory of an object based on prior movements.
• Appearance Preservation: Measures how consistently the object's appearance is rendered over time and from different viewpoints.

The Future of 4D Foundation Models

The implications of this study are vast—not just for the development of AI systems but for how they interact with the real world. The research emphasizes the need for improved training datasets that include more diverse scenarios where objects occasionally become occluded, alongside advancements in model architecture that could support true visual memory.

As AI continues to evolve, benchmarks like PERSISTBENCH will be crucial in guiding the technology towards a more robust understanding of memory—a fundamental aspect that has long been overlooked.

For those interested in the technical details or wishing to explore the dataset further, additional information can be found on the project's website.

Authors: Guangzhao He, Hadar Averbuch-Elor, Wei-Chiu Ma