Microsoft Research - World and Human Action Models towards gameplay ideation (Supplementary Video 1)
The study involved 27 game creatives and highlighted the importance of iterative tweaking in the creative process. Based on this, the generative model Wham was developed to support creative ideation by ensuring consistency, diversity, and persistency in gameplay sequences. Consistency ensures that generated sequences align with established game dynamics, as demonstrated by Wham's ability to maintain character and environment coherence over time. Diversity is shown through Wham's capability to generate multiple plausible sequences from a single starting point, capturing a wide range of human behaviors. Persistency allows for novel modifications to be integrated into the game state, such as adding new characters or objects, which Wham can adapt to and incorporate into the gameplay. The Weam demonstrator, a concept prototype, allows users to interact with Wham, generating diverse gameplay sequences from a single context frame, thus supporting creative ideation through divergent thinking and iterative tweaking.
Key Points:
- Wham supports creative ideation by ensuring consistency, diversity, and persistency in game sequences.
- Consistency is achieved by maintaining coherence with game dynamics and character behaviors.
- Diversity is demonstrated by generating multiple plausible sequences from a single starting point.
- Persistency allows for novel modifications to be integrated into the game state, enhancing creativity.
- The Weam demonstrator enables users to interact with Wham, fostering divergent thinking and iterative tweaking.
Details:
1. 🎮 Unlocking Creativity with Generative Models
- A user study with 27 game creatives highlighted the necessity of iterative tweaking in enhancing creativity.
- Key capabilities identified for generative models to aid creative ideation include consistency, diversity, and iterative improvement.
- Generative models offer significant potential in streamlining the creative process by providing diverse and consistent outputs that can be improved iteratively.
- The study found that using generative models can lead to more efficient creative workflows, reducing the time and effort needed for ideation.
2. 🎨 Ensuring Consistency in Game Dynamics
- The World and Human Action Model (Wham) is designed to generate gameplay sequences prompted by visuals or controller actions, ensuring they adhere to consistent game dynamics.
- Initially, Wham utilized 206 million parameters and, after 10,000 updates, achieved recognizable character movement and geometry, although trajectory consistency needed improvement.
- At 100,000 updates, Wham generated longer trajectories, yet faced challenges such as characters erroneously dropping to the ground when expected to fly, indicating a need for improved physics modeling.
- Upon reaching 1 million updates, Wham began accurately simulating behaviors and physics, such as correctly modeling flying mechanics, showcasing significant improvement in dynamics consistency.
- With further training using a 1.6 billion parameter model, Wham advanced in map geometry accuracy and character movement consistency, aligning generated visuals more closely with intended game dynamics.
- Technical improvements included refining physics engines and integrating more complex environmental interactions to address initial trajectory inconsistencies and improve overall model reliability.
3. 🌈 Embracing Diversity in Gameplay Paths
- Wham can generate diverse and plausible gameplay sequences from a single starting point.
- The model allows for three initial path choices: center, left, and right.
- Wham successfully simulates a variety of human behaviors and trajectories in gameplay.
- The diversity in paths demonstrates the model's ability to capture a wide range of gameplay styles.
4. 🔄 Achieving Persistency for Creative Control
4.1. Flexible Modifications to Game State
4.2. Introducing New Characters
4.3. Creative Interaction with Game Environment
5. 🚀 WEAM: Pioneering Creative Ideation in Gaming
- WEAM supports creative ideation in gaming through its demonstrator by generating multiple diverse gameplay sequences from a single promotional image, even though the image is different from the data WEAM is trained on.
- The WEAM demonstrator allows for the generation of diverse sequences by varying camera angles and user interface overlays, enhancing creative options for game developers.
- In one sequence, a character triggers a protective shield and the camera angle changes to reveal a staircase, showcasing WEAM's ability to create complex scenes from minimal input.
- The tool allows users to input controller commands to influence sequence generation, such as steering a character up stairs, which can be strategically used for ambush scenarios.
- WEAM enables users to introduce new elements, like enemy characters, by simply copying and pasting images into frames, thereby enhancing the action flow and providing more context for the sequences.
- The demonstrator illustrates WEAM's capacity to support Divergent thinking in creative processes, enabling the exploration of multiple action scenarios from a single image.