Open-Sora 2.0 - Open-source video AI with low training costs
The Singapore-based AI startup HPC-AI Tech has introduced a video AI model with lower training costs compared to other similar models. The startup called the new model Open-Sora 2.0.
The details
- According to the research paper, Open-Sora 2.0 needs about $200,000 for training costs. This is 5-10 times less than comparable models like Movie Gen or Step-Video-T2V.
- HPC-AI Tech used an affordable training pipeline with three main stages: (1) training a text-to-video model using low-resolution video data, (2) training an image-to-video model with low-resolution video data, and (3) fine-tuning an image-to-video model on high-resolution videos. The developers also saved computing resources by using pre-trained image models like Flux.
- Even though it has lower training costs, Open-Sora 2.0 performs just as well as leading video generation models like HunyuanVideo and Runway Gen-3 Alpha. In addition, Open-Sora 2.0 nearly matches the performance of OpenAI’s Sora.
- Open-Sora 2.0 can generate videos from text and images at resolutions up to 768×768 pixels for videos up to 5 seconds. The model is available on Hugging Face.
Our thoughts
This paper shows that high-quality video generation models can be developed with controlled costs through an optimized training strategy and pre-trained models.
However, Open-Sora has a lower resolution compared to OpenAI’s Sora, and its maximum length of five seconds is quite short. In comparison, OpenAI’s Sora can generate videos up to 60 seconds with a resolution of up to 1080p.
In addition, current diffusion models often cause unexpected problems, such as distorted objects and unnatural physical effects, which require further research.
More information: 🔗 GitHub | OpenSora Gallery | arXiv
Magic AI tool
Have you ever dreamed of turning your YouTube videos or podcasts into new content, e.g., social media posts? Then, CastMagic is right for you! It is a powerful tool for video creators and podcasters. It supports many languages (e.g., English, German, French, and many more).
In addition, you can also use it for meetings. Imagine you have a sales call. Then, you need a summary of the call, including the customer’s name, asked questions, next steps, and so on. Right? No problem! CastMagic can do all of this for you.
