Discover the groundbreaking video control technology by ByteDance, Boximator, a tool that lets you dictate the trajectory and dimensions of video elements with unprecedented precision through dual-box constraints.
How Boximator Works:
-
Dual-box Constraints: Utilize Hard Boxes to pinpoint the exact start and end positions and sizes of objects, establishing a clear composition framework. Apply Soft Boxes to suggest flexible movement boundaries, guiding objects smoothly between established hard box points.
-
Self-learning Approach: Boximator’s self-tracking capabilities allow the system to intuitively follow object movement, ensuring that predefined motions are articulated seamlessly, without the need for frame-by-frame user input.
-
Video Synthesis 101: The synergy of user inputs and Boximator’s predictive prowess culminates in videos that exhibit natural, user-defined object transitions and meets visual storytelling objectives.
Practical Application:
To illustrate, imagine crafting a video where a kitten leaps across a table:
- Use a Hard Box to mark the kitten’s tranquil beginning on one table end.
- Position a Hard Box on the opposite table end to define the jump’s destination.
- Implement Soft Boxes to articulate the jumping arc, ensuring a lifelike trajectory.
- Allow Boximator to animate the leap, fine-tuning with additional soft boxes if necessary for enhanced realism.
In essence, by blending Hard and Soft Boxes, users gain robust control over video object movements, adaptable to complexities ranging from a simple hop to intricate scenes.
Enhanced Base Model Capabilities:
Keeping the foundational video model weights intact, Boximator enhances object motion control while preserving original quality and knowledge—a fusion empowering more control and application breadth.
Wide-Ranging Integration:
Designed as a plug-in, Boximator seamlessly adapts to diverse video diffusion models, broadening its utility across varied creative challenges.
Check out the complete research here. Stay tuned for the GitHub release!
Boximator: Bring Fine-grained Motion Controllability to Video Synthesis | Bytedance Research