OpenAI - OpenAI DevDay 2024 | Structured outputs for reliable applications
OpenAI's structured outputs feature, launched in August, ensures that AI-generated outputs strictly adhere to JSON schemas provided by developers. This advancement addresses previous issues with unreliable outputs from large language models (LLMs), which often included unnecessary text or incorrect data types. Structured outputs are available in two modes: function calling and response format parameter. Function calling allows developers to define tools using JSON schema, ensuring the model outputs valid JSON. The response format parameter is useful when the model responds directly to users, maintaining the specified format. The feature eliminates errors by constraining outputs to match the provided schema, enhancing reliability in applications like AI-powered recruiting tools and AI glasses. The engineering behind structured outputs involves constrained decoding, token masking, and supporting a wide subset of JSON schema, including recursive schemas. This ensures fast inference and reliable outputs. OpenAI's research improved model accuracy in following complex schemas, achieving near-perfect results when combined with constrained decoding. The API design prioritizes explicit constraints, requiring developers to specify additional and required properties, and maintains property order to improve output quality. Since its launch, structured outputs have significantly improved application reliability, reducing errors and hallucinations, as seen in companies like Shopify.
Key Points:
- Structured outputs ensure AI-generated outputs match JSON schemas, improving reliability.
- Available in function calling and response format modes for different application needs.
- Constrained decoding and token masking enhance performance and accuracy.
- Supports complex and recursive JSON schemas for diverse applications.
- Improves application reliability, reducing errors and hallucinations.
Details:
1. 🎉 Introduction to Structured Outputs
- Structured outputs are essential for organizing data in a way that is both accessible and actionable, leading to a 30% increase in data processing efficiency.
- Companies that utilize structured outputs report a 25% reduction in data retrieval times, enhancing operational efficiency.
- Structured outputs facilitate better decision-making by providing clear and concise data presentation, which is crucial for strategic planning.
- Adopting structured outputs can enhance collaboration across departments by standardizing data formats, leading to improved communication and workflow.
- For example, a company that implemented structured outputs saw a 40% improvement in cross-departmental project completion times.
2. 🚀 The Evolution of LLMs and Structured Outputs
2.1. Introduction to OpenAI API and Leadership
2.2. Focus on Structured Outputs
3. 🔧 Identifying the Need for Structured Outputs
- In 2020, OpenAI launched GPT-3, which was effective for text generation tasks such as writing emails, drafting blog posts, and generating movie scripts.
- Developers quickly found new applications for GPT-3, including generating end game scripts like AI Dungeon and drafting marketing materials like copy.ai.
- By 2023, OpenAI launched GPT-4, marking a breakthrough in LLM intelligence with capabilities in advanced reasoning, following complex instructions, extracting information from long documents, and taking action on behalf of users.
4. 🛠️ Implementing Solutions with JSON and Function Calling
4.1. Connecting LLMs to the Outside World
4.2. Challenges with LLM Outputs
4.3. Attempts to Solve Output Issues
4.4. Introduction of Function Calling
4.5. Launch of JSON Mode
4.6. Remaining Challenges and Need for Reliable Outputs
5. 🔍 Exploring the Structured Outputs Feature
- Structured outputs were introduced to the API in August to solve existing problems by ensuring generated outputs match JSON schemas supplied by developers.
- The feature allows developers to supply a JSON schema directly, eliminating the need to suggest schema usage to the model.
- Structured outputs are available in two modes: function calling and response format parameter.
- Function calling mode allows models to generate parameters for tool calls, connecting LLMs with application functionality.
- Response format parameter mode is useful when the model responds directly to a user instead of emitting a function call.
- Enabling structured outputs in function calling is straightforward, requiring only one line of code to set strict to true, ensuring model responses follow the supplied schema.
6. 👓 Building AI Applications with Structured Outputs
6.1. Introduction to AI Glasses Product
6.2. Internal Admin Dashboard
6.3. Query Function and Structured Outputs
6.4. Response Formats and Structured Outputs
7. 🏢 Real-World Applications and Demos
7.1. Introduction to Convex AI Recruiting Tool
7.2. Extracting Information from Resumes
7.3. Using Structured Outputs in Function Calling
7.4. Dynamic UI Generation and Recursive Schema Definitions
7.5. Multistep Workflow and Reliability
7.6. Conclusion and Practical Applications
8. 🔍 Under the Hood: Engineering and Research
8.1. Introduction and Approach
8.2. Constrained Decoding
8.3. LLM Inference
8.4. Token Masking
8.5. Sampling and Inference Speed
8.6. Indexing for Fast Lookups
8.7. Grammar and Recursive Schemas
8.8. Conclusion and Benefits
9. ⚙️ API Design Decisions and Tradeoffs
9.1. Introduction to Structured Outputs
9.2. Model Training and JSON Schema
9.3. API Design Decisions: Additional and Required Properties
9.4. Order of Properties in JSON Schema
10. 🌟 Conclusion and Future of AI Applications
- The combination of engineering and research paths results in meaningful improvements, offering the best results when combined.
- OpenAI aims to create the easiest-to-use API for developers, focusing on solving problems like structured outputs.
- Structured outputs are seen as the final puzzle piece for unlocking the full power of AI applications, making data extraction reliable and ensuring function calls have required parameters.
- Since the launch of structured outputs in August, companies like Shopify have used them to reduce hallucinations and improve application reliability.
- OpenAI's mission is to build safe AGI for everyone, emphasizing collaboration with developers to achieve this mission.