OpenAI - OpenAI o3 and o3-mini—12 Days of OpenAI: Day 12
OpenAI has introduced two new AI models, 03 and 03 Mini, which are designed to perform complex reasoning tasks. These models are not yet publicly available but are open for public safety testing. The 03 model shows significant improvements in technical benchmarks, achieving 71.7% accuracy on software tasks and outperforming previous models in coding and mathematics competitions. It also excels in PhD-level science questions and has set a new state-of-the-art score on the ARC AGI benchmark, indicating progress towards general intelligence. The 03 Mini model offers cost-efficient reasoning capabilities and supports adaptive thinking time, allowing users to adjust reasoning effort based on task complexity. Both models are part of OpenAI's efforts to enhance AI safety and performance through public testing and new safety techniques like deliberative alignment, which improves the model's ability to identify safe and unsafe prompts.
Key Points:
- 03 and 03 Mini models focus on complex reasoning tasks.
- 03 achieves 71.7% accuracy on software benchmarks, 96.7% on math tests.
- 03 sets a new record on ARC AGI benchmark, indicating AI progress.
- 03 Mini offers cost-efficient reasoning with adjustable thinking time.
- Public safety testing is open to researchers to enhance model safety.
Details:
1. 🚀 Launching the Next Frontier Model
- The event marks the launch of the first reasoning model, 01, which has been available for 12 days.
- The model is designed to handle increasingly complex tasks requiring significant reasoning, setting a new standard in AI capabilities.
- This launch is considered the beginning of a new phase in AI development, with potential to significantly impact various industries.
2. 🔍 Introducing Models 03 and 03 Mini
- Two new models, 03 and 03 Mini, are being announced, marking a significant addition to the product lineup.
- The naming convention deviates from logical sequence, skipping 'O2' to 'O3', which is part of the company's tradition of unconventional naming strategies.
- This approach reflects the company's innovative mindset and willingness to break from traditional norms, potentially appealing to a market that values creativity and uniqueness.
3. 🛡️ Public Safety Testing Announcement
3.1. 🛡️ Public Safety Testing Announcement
3.2. Model Capabilities and Demonstrations
4. 💻 O3's Technical Capabilities and Benchmarks
- O3 achieves 71.7% accuracy on Sweet Bench Verified, a benchmark for real-world software tasks, outperforming O1 models by over 20%.
- On Codeforces, a competitive coding platform, O3 attains an ELO of 2727 under high test time compute settings, far exceeding the O1 model's ELO of 1891.
- O3's ELO score of 2727 surpasses the personal best of 2500 by a competitive programmer and even exceeds the chief scientist's score at OpenAI.
5. 📊 Advancements in Mathematical and Scientific Benchmarks
- The model achieves 96.7% accuracy on competition math benchmarks, compared to 83.3% for the previous version (01).
- On the GPQ Diamond benchmark, which measures PhD-level science questions, the model scores 87.7%, a 10% improvement over the previous 78% performance.
- Expert PhDs typically score around 70% in their field of strength, highlighting the model's advanced capabilities.
- There is a need for harder benchmarks as current models are nearing saturation in existing tests.
- Epic AI's Frontier Math Benchmark is considered the toughest mathematical benchmark, with current models achieving less than 2% accuracy on it.
6. 🏆 Breaking New Ground with ARC AGI Benchmark
- The ARC AGI Benchmark, established in 2019, remained unbeaten for 5 years, representing a significant challenge in AI development.
- The benchmark tests AI's ability to understand transformation rules from input to output examples, a task that is straightforward for humans but difficult for AI.
- ARC AGI tasks require models to learn new skills dynamically rather than relying on memorized tasks, testing adaptability and learning capabilities.
- Version 1 of ARC AGI saw a slow progression from 0% to 5% over 5 years with leading models.
- A new model, 03, achieved a state-of-the-art score of 75.7 on ARC AI's semi-private holdout set, verified under low compute settings.
- This achievement places the model as the new number one entry on the ARC AGI public leaderboard, meeting the compute requirements for public ranking.
7. 🤝 Collaboration with ARC Prize Foundation
- AI model O03 achieved a score of 87.5% on a hidden holdout set, surpassing the human performance threshold of 85%, marking a significant milestone in AI capabilities.
- This achievement represents new territory in the RCGI world, as no system or model has previously reached this level of performance.
- The collaboration aims to develop enduring benchmarks like Arc AGI to measure and guide AI progress, with plans to partner with OpenAI to create the next frontier benchmark.
- The ARC Prize Foundation will continue its initiatives in 2025, with more information available at ARC pri.org.
8. 🧠 Introducing O3 Mini and Its Capabilities
- O3 Mini is a new model in the O3 family, designed to be a cost-efficient reasoning model with strong capabilities in math and coding.
- The model supports adaptive thinking time with three options: low, medium, and high reasoning effort, allowing users to adjust based on their needs.
- In coding evaluations, O3 Mini outperforms O1 Mini, achieving better performance with median thinking time at a fraction of the cost.
- O3 Mini's high reasoning effort is only a few hundred points away from top performance benchmarks, offering significant cost-to-performance gains.
- The model demonstrates a new cost-efficient reasoning frontier, achieving better performance than O1 Mini at a lower cost.
- O3 Mini supports function calling, structured outputs, and developer messages, providing a cost-effective solution for developers.
- In math evaluations, O3 Mini achieves comparable or better performance than O1 Mini, with reduced latency nearly matching GPT-4's instant response times.
- The model's low reasoning effort drastically reduces latency, achieving near-instant response times comparable to GPT-4.
- O3 Mini's API features include support for function calling and structured outputs, enhancing developer experience.
- The model's performance in evaluations shows it as a more cost-effective solution, achieving better results at a lower cost.