No Priors: AI, Machine Learning, Tech, & Startups - No Priors Ep. 96 | With Modal CEO and Founder Erik Bernhardsson
Eric Bernhardson, founder of Modal, shares his journey from Spotify to creating a cloud platform designed to simplify AI and machine learning infrastructure. Modal aims to make cloud development as seamless as local development by eliminating the complexities of traditional cloud tools like Docker and Kubernetes. The platform offers a large compute pool with thousands of GPUs and CPUs, enabling users to access resources on demand without long-term commitments. This flexibility is particularly beneficial for startups and companies needing scalable GPU access for tasks like inference and training. Modal's Python SDK allows developers to write code that is automatically turned into serverless functions, simplifying the deployment process. The platform gained significant traction with the rise of stable diffusion models, offering easy GPU access for AI-generated content across various modalities, including audio and video. Bernhardson also discusses the broader implications of AI infrastructure, emphasizing the need for more flexible and efficient GPU usage and the potential for AI-native data storage solutions.
Key Points:
- Modal simplifies AI and machine learning infrastructure by providing a cloud platform that feels like local development.
- The platform offers on-demand access to a large compute pool, eliminating the need for long-term GPU commitments.
- Modal's Python SDK turns code into serverless functions, easing deployment and infrastructure management.
- The platform gained traction with stable diffusion models, supporting AI-generated content in various formats.
- Eric Bernhardson highlights the potential for AI-native data storage solutions and more efficient GPU usage.
Details:
1. 🎙️ Welcome and Guest Introduction
- Eric Bernhardson is the founder and CEO of Modal, a serverless cloud platform tailored for AI, machine learning, and data applications.
- With a strong background, Eric previously worked at Better.com and Spotify, where he spearheaded machine learning initiatives and developed a recommender system at Spotify.
2. 👨💻 Eric's Background and Spotify Experience
2.1. Eric's Contributions at Spotify
2.2. Eric's Leadership at Better.com
2.3. Founding Modal
3. 🚀 Founding Modal: Vision and Challenges
3.1. Vision and Initial Challenges
3.2. Current Offerings and Impact
4. 🧩 Addressing Cloud Infrastructure and GPU Use
- Current GPU use is often wasteful in the industry, highlighting the need for more flexible capacity.
- GPUs are expensive, and access typically requires long-term commitments, which are not ideal for startups.
- Cloud infrastructure should provide on-demand GPU access, similar to CPU access, to support dynamic needs.
- Many companies face challenges with capacity planning, leading to under or over-provisioning of GPU resources.
- Modal offers a usage-based model, charging only for the time containers are running, which accommodates variable workloads effectively.
- The shift to inference demands more flexibility in GPU usage as needs are unpredictable and volatile.
- Modal handles bursty and unpredictable workloads by pooling resources across multiple customers.
- There is an interest in supporting shorter, experimental training runs, which require flexible capacity.
5. 🔗 Modal's Comprehensive Platform Strategy
- Modal's platform aims to increase engineer productivity by covering end-to-end machine learning use cases, initially focusing on inference as a 'killer app.'
- The platform now aims to support the entire machine learning lifecycle, including data preprocessing, training, and feedback loops, seamlessly integrated into one ecosystem.
- Customers utilize Modal for extensive batch preprocessing tasks, such as feature extraction on petabytes of video data using GPUs, before performing training processes elsewhere and returning for inference.
- Modal is strategically expanding to support training processes, thereby facilitating a comprehensive solution for the full machine learning lifecycle.
- The platform automates data pipelines and tasks, such as nightly batch jobs, to streamline operations and enhance efficiency.
6. 🌐 Navigating Cloud Adoption and Security
- Modal differentiates itself by adopting a cloud-maximalist approach, focusing on building a multi-tenant platform that efficiently manages capacity.
- The platform offers instantaneous access to hundreds of GPUs, enabling bursty computational tasks with ease.
- Modal's infrastructure allows running custom user code safely in containers, addressing complex containerization challenges.
- The company developed its own scheduler, container runtime, and file system to quickly boot containers, supporting a general-purpose platform.
- Unlike other vendors focused on specific applications like inference or LMS, Modal aims to build a broad and adaptable platform capable of supporting various products.
7. 📈 Trends in Open Source and AI Development
- Many large enterprises have adopted cloud platforms such as Azure, GCP, or AWS, incentivized by marketplace credits and security reviews. This shift reflects a broader trend towards leveraging cloud infrastructure to enhance operational efficiency.
- Concerns about cloud adoption include latency issues and increased dependency on third-party services. Enterprises are weighing these against the benefits of using their existing cloud providers or hyperscalers.
- Security focus has shifted from the network layer to the application layer, aligning with decreasing bandwidth costs, which make cloud solutions more attractive.
- Snowflake's success story highlights the viability of the infrastructure as a service model, indicating a growing acceptance of multi-tenant solutions among enterprises.
- Cost-effective strategies like storing data in services with zero egress fees, such as R2, help manage bandwidth transfer costs effectively.
- The multi-tenant model offers improved capacity management, despite initial resistance from enterprises, showcasing a strategic shift towards more efficient resource utilization.
8. 🔍 AI Models and Emerging Modalities
8.1. AI Compute Resource Management
8.2. Open Source and Proprietary Models
8.3. Emerging Focus Areas in AI
9. 🛠️ Infrastructure Needs: Storage and Efficiency
- AI infrastructure currently lacks efficient tools for engineers to run custom workflows and train models, necessitating solutions like Modal to simplify these processes.
- There is high demand for advanced vector databases and efficient storage solutions for training data, presenting opportunities for innovation.
- Training large models is costly and time-consuming due to complex networking setups, which could be alleviated by reducing these requirements and using dispersed GPUs across data centers.
10. 🔍 Vector Databases and AI Storage Solutions
- There is a significant debate about the necessity of standalone Vector databases compared to using enhanced traditional databases like Postgres with PG Vector, which might suffice for current needs.
- Future developments in AI could necessitate AI-native data storage solutions that differ in form and interface from traditional databases, potentially allowing direct input and querying of diverse data types such as text, video, and images without relying solely on vector embeddings.
- The market is witnessing a shift where AI-driven data storage solutions could provide more seamless integration and efficiency in handling complex data forms.
11. 🤔 AI's Role in Software Development and Physics
- Companies prioritizing model quality should train their own models to gain a competitive edge, as custom models can outperform off-the-shelf solutions.
- Off-the-shelf models may restrict a company's ability to claim superior solutions, affecting their competitive moat.
- Custom models often provide a unique competitive advantage, particularly in domains such as audio, video, and image processing.
- For example, in audio and image processing, custom models have been known to enhance performance significantly, leading to better product differentiation.
- However, developing custom models requires substantial resources and expertise, highlighting the need for strategic investment decisions.