Google unveils Lumiere – generative AI for video

Just like many others, Google is heavily investing in AI, and they have now announced that it's time to take the next step and develop the possibilities for generative AI in the realm of motion – video that is created from descriptive text.

Google Research has released a document describing the development of Lumiere, and in conjunction with this, they have presented a demo that shows how text can create objects that both move and change, such as animals that change color and appearance, but also people changing color and clothing style. 

Everything is described through a prompt where one enters text for what the video should contain, which means it involves so-called generative AI since the model does not start from any user-inputted templates but is created by the model itself – although it is also possible to upload an image to make it move.

Exactly which material the model has been trained on is not disclosed, but the document by Google states that 30 million videos have been used: 

“We train our T2V [text to video] model on a dataset containing 30M videos along with their text caption. The videos are 80 frames long at 16 fps (5 seconds). The base model is trained at 128×128.”

Camera & Image has previously written about the creation of an 8-minute long film about the rainforest using the Stable Diffusion model, which displays birds, water, plants, and vegetation with high detail reproduction.

In addition, the models for regular images have also quickly improved, as seen in Adobe's newly released Firefly Image 2 with more lifelike images of people. At the same time, Getty Images has also invested in generative AI by offering it in their image service "Generative AI by iStock".

The short film about the rainforest created entirely with AI – are you fooled?

Adobe releases Firefly Image 2: AI engine that is better at humans