We are working on a new method tentatively called “Visual Task Planning,” or VTP, which involves using deep neural networks to hallucinate futures so robots can understand the consequences of their environment and explore many different possible futures.

The video above shows the predictive model used in Visual Task Planning as applied to a set of simulation data. It hallucinates possible worlds after following multiple high-level actions: think of it as imagining multiple possible futures, such as “what will happen if I try to grab the red block?”

VTP is all about giving robots prospection. This is how humans solve problems in complex environments: we imagine prospective futures and see what will happen. I am excited about following up on this in future work.