OpenAI's Sora video generation model can also render video games

Image credits: OpenAI

The new OpenAI – and the first! — The video production model, Sora, can achieve some truly amazing cinematic work. But the model is balanced more It could be OpenAI at first, at least judging by the technology paper Posted this evening.

The paper, titled “Video Generation Models as Universal Simulators,” co-authored by a group of researchers at OpenAI, pulls back the curtain on key aspects of Sora's architecture — for example, it reveals that Sora can generate videos with arbitrary resolution and aspect ratio ( up to 1080p). According to the paper, Sora is capable of performing a range of photo and video editing tasks, from creating looping videos to stretching videos forward or backward in time and even changing the background in an existing video.

But what's most interesting to this writer is Sora's ability to “simulate digital worlds,” as OpenAI co-authors put it. In one experiment, OpenAI set Sora loose on Minecraft and had him render the world — and its dynamics, including physics — while simultaneously controlling the player.

Sora controls a player in Minecraft – depicting the video game world as he does so. Note that the graininess was provided by the video to GIF converter, not by Sora. Image credits: OpenAI

So how can Sora do this? like that Notice Written by Nvidia Senior Researcher Jim Fan (via quartz), Sora is more of a “data-driven physics engine” than a creator, too. It's not just creating a single image or video, but determining the physics of every object in the environment – ​​and presenting an image or video (or an interactive 3D world, as the case may be) based on these calculations.

“These capabilities suggest that the continued expansion of video modeling is a promising path toward developing highly capable simulators of the physical and digital world, and the objects, animals, and people who live within them,” the co-authors wrote.

Now, Sora's usual video game limitations apply. The model cannot accurately approximate the physics of basic interactions such as shattering glass. Even with interactions like that Can The model, Sora is often inconsistent – for example depicting someone eating a burger but failing to leave bite marks.