An image comes to life, a description becomes a scene, the style of an image drives an entire video. The new Google Lumiere, at least in the tests released by the Cupertino company itself, demonstrates once again that AI knows how to amaze us. The new tool, launched in collaboration with the Weizmann Institute of Science and Tel Aviv University, allows the creation of videos using photos and text instructions. But also to edit and apply different styles to existing videos.
Google Lumiere, AI creates realistic videos from images and text prompts
The project, of which you can see some examples in the video, is called Lumiere to pay homage to the Lumière brothers, French pioneers of cinema, inventors of the movie camera and the cinema projector. And Google expects its new tool to be just as revolutionary.
Unlike programs like Stable Diffusion, which often show imperfections in the consistency of animations, Lumiere IA has introduced an architecture called “Space-Time U-Net”. This space-time network allows you to Generate the entire video in one stepavoiding the creation of intermediate sequences that could cause inconsistencies in videos.
The Space-Time U-Net architecture analyzes several proposed movements, choosing the most plausible to ensure a coherent and smooth video. In this way, you can insert a photo of an object, such as a stuffed animal, and ask the program to make it walk from point A to point B. The result is a space-time sequence in which each activity isclosely related to the others. This makes videos smoother.
A huge database, and many features
Lumiere AI has a database yes 30 million videos along with related text captions. This machine learning approach allowed the model to gain a broad understanding of visual context and connections between actions and textual descriptions. In this way, it can generate videos starting from a text or a photograph. But it can also understand the style of an image (for example, a “neon” or “impressionistic” effect) and apply it to a video, whether generated or edited.
Currently, Lumiere is not available as open software to the public, but it represents an experimental research project. So we can’t use it to create videos — and that means you can’t use it for either creare video deep fake.
In all likelihood, before releasing it Google will not only refine the software, but will also work with universities to avoid any abuse. Or at least, that’s what we expect: we should all have understood that generative AI is no joke. Even when, as we see in the Google video, it creates videos of dancing bears.
