OpenAI - OpenAI DevDay 2024 | Community Spotlight | DataKind
Caitlyn Augustine from DataKind highlights the critical need for high-quality data in humanitarian efforts, noting that 300 million people require assistance globally, with a $46 billion funding gap. DataKind collaborates with humanitarian organizations to identify data access challenges and explores solutions like generative AI for metadata prediction. Despite the existence of metadata standards like hexel, adoption is low, leading to interoperability issues. Generative AI can improve metadata tagging accuracy, with recent tests achieving over 95% accuracy for common metadata like locations and dates. The project aims to make data processing cost-effective and efficient, allowing humanitarian organizations to handle large data volumes with minimal resources. The initiative is part of a broader effort to create a comprehensive humanitarian data system, including an AI assistant for rapid data access and response.
Key Points:
- Generative AI can significantly improve metadata tagging accuracy, achieving over 95% accuracy for common data types.
- DataKind's initiative aims to address the $46 billion funding gap in humanitarian aid by enhancing data quality and accessibility.
- The project targets a 70% accuracy rate for metadata tagging, with a cost-effective solution allowing processing of 100 tables weekly for $5.
- The AI system is designed to integrate seamlessly into existing workflows, reducing manual data correction efforts.
- The broader goal is to develop a comprehensive humanitarian data system, including an AI assistant for rapid, accurate data access.
Details:
1. 🌍 Introduction to DataKind's Mission
1.1. DataKind's Mission
1.2. Key Personnel
2. 📊 The Humanitarian Data Challenge
- There is an enormous need for timely and high-quality data in the humanitarian space, which is crucial for effective response and resource allocation.
- Currently, 300 million people worldwide require humanitarian assistance, underscoring the vast scale of the challenge.
- There are 40 coordinated global appeals addressing these needs, indicating a structured approach to tackling the issues.
- Despite these efforts, there is a $46 billion gap in funding for these humanitarian efforts, highlighting a significant shortfall that needs to be addressed to meet global needs effectively.
3. 🚀 Innovations in Data Utilization
- The UN's interactive dashboard for Afghanistan integrates data from local governments, NOS, and UN teams, enabling rapid disaster response.
- This dashboard allows responders to quickly identify disaster locations and deploy appropriate interventions, showcasing efficient resource utilization.
- Despite its success, such high-quality data integration remains an exception rather than the norm, highlighting the need for broader adoption to save lives.
- Challenges in data integration include varying data standards and limited technological infrastructure in some regions, which need addressing for broader implementation.
- The success of the dashboard underscores the potential for data-driven solutions to enhance disaster response, emphasizing the importance of overcoming integration challenges.
4. 🔍 Tackling Metadata Challenges
4.1. Metadata Challenges in Humanitarian Data
4.2. Solutions for Metadata Challenges
5. 🤖 Leveraging AI for Metadata Solutions
- Approximately 50% of metadata tagging is incorrect or non-standard, making it unfit for purpose.
- Generative AI is being explored to improve metadata tagging, building on a proof of concept from 5 years ago that faced implementation challenges.
- Using AI models like GPT, the tagging process has been expanded to cover a broader knowledge base with less friction.
- The initiative began in 2023 and expanded in 2024, with the last testing round completed in August, involving three different models and prompting approaches.
- Only 25% of datasets currently have accurate metadata, but stakeholders prioritize improvement over perfection, aiming for more right than wrong.
- A 70% accuracy target was set based on literature indicating meaningful results at this level in similar contexts.
- The solution is designed for humanitarians and nonprofits, with a cost target of $5 per week to process around 100 tables, aligning with their budget constraints.
- The workflow aims for a processing time of 1 second per table, totaling about an hour for 100 tables, integrating seamlessly into existing workflows.
6. 🧠 Training and Testing AI Models
6.1. Data Enrichment and Preparation
6.2. Model Testing and Accuracy
7. 🔄 Enhancing AI Accuracy and Efficiency
- Initially, avoiding fine-tuning by directly prompting for hexel tags and attributes was considered effective but did not align with hexel standards.
- Incorporating specific instructions and rules in prompts significantly improved alignment with hexel data standards.
- The revised prompting strategy successfully met accuracy, time, and cost targets, unlocking thousands of variables for humanitarian use.
- Ongoing improvements and distillation efforts are further enhancing AI capabilities, ensuring continuous progress.
8. 🤝 Future Directions and Humanitarian AI Assistant
- The metadata prediction is a component of a larger humanitarian data project system, indicating a modular approach to data management.
- The system is designed to provide humanitarians with rapid access to high-quality, timely data, enhancing their ability to respond quickly to crises.
- The humanitarian AI assistant integrates harmonized, interoperable data, allowing users to interact via chat to obtain ground truth verified information, facilitating rapid response efforts.
- The development of the AI assistant has been a collaborative effort with humanitarians, ensuring the tool meets the practical needs of its users.