Jeff Su: The video provides insights into the role of a Product Marketing Manager at Google, focusing on strategies for increasing product adoption and tips for non-technical interviews.
Weights & Biases: The video discusses building agentic AI applications using Weights and Biases' Weave tool, focusing on autonomous AI agents and their workflows.
Jeff Su - A Day in My Life at Google (8 years in)
The speaker, a Product Marketing Manager (PMM) at Google, explains their role in driving product adoption in local markets, using Google Gemini as an example. They describe the structure of marketing teams at global, regional, and local levels, emphasizing the autonomy local marketers have in choosing strategies to increase monthly active users. The speaker outlines a campaign process, including identifying market opportunities, targeting audiences, and choosing tactics based on career goals. They also share a project management tip involving three essential documents: project notes, working spreadsheets, and briefing slides, each serving different audiences. Additionally, the speaker provides tips for non-technical interviews, highlighting the importance of demonstrating thought processes and using the XYZ bullet point structure in resumes. They recommend using Google's interview warm-up tool for preparation.
Key Points:
- Local marketers have autonomy in strategy selection to increase user engagement.
- Campaigns should be tailored to market opportunities and target audiences.
- Project management involves using specific documents for different audiences.
- Non-technical interviews focus on thought processes, not just answers.
- Use Google's interview warm-up tool for effective interview preparation.
Details:
1. 🎉 Enthusiasm for Innovation
1.1. Culture of Innovation
1.2. Strategic Initiatives and Challenges
2. 😂 Humor and Early Birds
2.1. Humor in the Workplace
2.2. Benefits of Early Work Hours
3. 🧑💼 Inside the Life of a Google PMM
3.1. Global PMM Responsibilities and Strategies
3.2. Local PMM Execution and Strategies
4. 🍽️ Culinary Perks and Audience Engagement
4.1. Culinary Perks
4.2. Audience Engagement
5. 💰 Earnings and Work-Life Dynamics
- Current earnings are over $11,000 a month before tax, excluding equity and bonuses, indicating strong income potential in certain markets.
- Asian markets offer relatively lower pay compared to the US and Western Europe, impacting the earnings and potentially the lifestyle of professionals in these regions.
- Software engineers in the US and Western Europe earn significantly more than their counterparts in Asian markets, highlighting regional disparities in compensation.
- The income disparities between regions can affect lifestyle choices and work-life balance, as professionals in higher-earning regions might have more financial flexibility.
- Considering the cost of living variations, the real value of earnings can differ greatly, affecting purchasing power and lifestyle quality across regions.
6. 💼 Intense Work Culture and Career Insights
6.1. Personal Work Experience
6.2. Asian Work Culture
6.3. Cultural Transition and Work Environment
7. 📅 Navigating Daily Routines
- Future content will focus on structuring daily and weekly routines to enhance productivity, suggesting insights into effective time management.
- The speaker plans to share specific routine strategies, which could include techniques for prioritizing tasks, setting goals, and managing time blocks effectively.
8. 🗂️ Project Management Mastery
- Every project requires three essential documents: project notes, working spreadsheet, and briefing slides.
- Project notes capture all project-related discussions, decisions, and history, primarily for the project owner's use, providing a comprehensive record of the project's evolution.
- The working spreadsheet acts as the central collaboration hub for the core project team, detailing timelines, milestones, action items, and resources, thereby facilitating day-to-day project management and ensuring alignment among team members.
- Briefing slides offer a high-level visual overview for stakeholders outside the core team, focusing on key status updates and strategic objectives without operational details, thus facilitating informed decision-making.
- These documents interact by ensuring seamless communication and alignment across different levels of project engagement, with project notes feeding into the working spreadsheet and key highlights being distilled into briefing slides for broader stakeholder communication.
9. 🤝 Effective Interview Prep and Wrap-Up
- Interviewers evaluate candidates on four key attributes: general cognitive ability, role-related knowledge, googliness, and leadership. Candidates should tailor their preparation to demonstrate strengths in these areas.
- For general cognitive ability, focus on case interview questions common in consulting, emphasizing the thought process over finding the correct answer. Recommended resources include Victor Chang's 'Case Interview Secrets' and 'Case in Point.'
- Resumes should follow the XYZ bullet point structure to clearly demonstrate impact and achievements, e.g., 'Increased customer satisfaction by 22% in 3 months by designing a new onboarding process.'
- Leverage Google's interview warm-up tool for conducting mock interviews and practicing industry-specific questions to build confidence.
- Utilize Google Gemini for role-playing as an interviewer, helping to gain insights into the interview process and improve interview readiness.
Weights & Biases - Building agentic AI applications with W&B Weave
The discussion begins by defining AI agents as autonomous systems that operate independently, executing multi-step tasks in a cyclical manner. These agents use memory and tools to enhance their capabilities and work towards achieving their end goals. A practical example is provided through a financial analyst dashboard, where an AI agent analyzes stock market data to provide insights. The workflow involves generating search queries, summarizing content, and verifying reports, showcasing the complexity of agentic workflows.
Weave is introduced as a tool that captures all data related to AI agent development, including inputs, outputs, metadata, and code. It helps in optimizing AI agent performance across various dimensions such as quality, latency, cost, and safety. Weave provides interactive trace trees, a flexible framework for evaluation, and guardrails for protection. The tool supports every step of the agent development workflow, from prototyping to deployment, and offers features like evaluation comparisons and verification agents to ensure accuracy and reliability.
Key Points:
- AI agents are autonomous systems executing multi-step tasks independently.
- Weave captures and organizes data for AI agent development, aiding in optimization.
- The financial analyst dashboard example illustrates agentic workflows in action.
- Weave provides tools for prototyping, testing, and evaluating AI agents.
- Verification agents ensure the accuracy and reliability of AI-generated reports.
Details:
1. 👋 Introduction to Agentic AI Applications
- Agentic AI applications are autonomous systems that operate without human oversight, enabling them to function independently.
- These systems can plan and execute multi-step tasks in a cyclical manner rather than a linear one, adapting to the requirements of the task.
- Agents utilize memory and tools to enhance their capabilities, allowing them to perform complex tasks autonomously.
- A key characteristic of agentic AI is its focus on achieving end goals, working continuously until the goal is met.
- An illustrative workflow involves the AI agent interacting with an end user, generating a multi-step plan in response to a query, and executing the plan until completion.
- For example, a customer service agentic AI can autonomously handle inquiries by assessing the query, planning responses, and following up without human intervention, improving efficiency by 30%.
- In logistics, agentic AI systems optimize delivery routes in real-time, reducing fuel consumption by 20% and improving delivery times by 15%.
2. 📈 Demo: Financial Dashboard & Market Analysis
2.1. Financial Dashboard Features
2.2. Market Analysis Insights
3. 🔍 In-Depth Look at Agent Workflows
- Analysts initiate the process by entering a research topic into the AI interface, which triggers a structured workflow.
- The planner agent develops a strategic plan comprising 5 to 15 search engine queries to identify relevant online data.
- After data collection, a financial writer agent synthesizes the information with support from fundamentals and risk analyst agents.
- To ensure accuracy, a verification agent reviews the summarized report before it is published on the dashboard.
4. 🛠️ Weave: Capturing and Analyzing Data Effectively
- Weave captures comprehensive data including inputs, outputs, metadata, and code during AI development, evaluation, and optimization.
- Aggregated metrics such as total tokens, total cost, latency, and trace size are available for analysis.
- The comprehensive trace tree visualizes the input and output data for each agent and step in the workflow, aiding in detailed examination of LLM API calls.
- The default view of the trace tree shows task completion by each agent, with a timeline slider for task execution review.
- Code composition view organizes traces by nesting within the agent, allowing quick visualization of workflow execution and related tasks, with aggregated metrics like task count and latency.
- The flame view is an interactive chart that visualizes the agentic workflow, allowing navigation and detailed examination of tasks, identifying concurrent activities, bottlenecks, and dependencies.
5. ⚙️ Challenges in Building Reliable AI Agents
- Building a multi-step workflow with LOM calls is straightforward, but deploying a reliable agent in production is complex due to non-deterministic behavior, leading to inconsistent responses.
- Agents are prone to making mistakes that humans would not make, primarily because the underlying engine (LMS) is non-deterministic, resulting in unpredictability in responses.
- Agentic workflows are complex and opaque, making it hard to understand task interdependencies, durations, and critical relationships needed for production-ready applications.
- Weave simplifies the productionization of agents by tracking and organizing all inputs, outputs, code, and metadata, enabling rapid iteration and optimization across quality, latency, cost, and safety.
- Weave provides a playground for prototyping and testing, with interactive trace trees and a flexible framework for rigorous evaluation, ensuring the development of reliable enterprise-grade AI agents.
- Guardrails are integrated to protect brand integrity, agent functionality, and end-users.
6. 🚀 Prototyping AI Agents with Weave
- Weave significantly optimizes the prototyping phase of AI agent development by enhancing both the iterate and deploy steps within the workflow.
- The platform provides a structured environment for creating the first version of an AI agent, effectively streamlining the initial development process and reducing the time to market.
- Utilizing Weave, developers can efficiently test and refine agent capabilities, particularly in complex fields like financial research, ensuring robust initial models.
- Case studies highlight a reduction in development time by up to 30% when using Weave, demonstrating its practical impact on the workflow.
7. 🧪 Task Evaluation and Optimization
- The financial research agent begins by generating search queries to conduct the necessary research for a financial report.
- A notebook with Python code is used to initiate this task as part of an agentic workflow, which involves an LLM call to request queries.
- These queries are assigned to individual financial search agents to perform searches and retrieve relevant information.
- A single line of code initializes 'weave' and a 'weave.op' decorator is used to enhance collected trace data.
- After executing the call, the process moves to 'weave' to view the call on the traces page and generate search queries.
- The system allows for further investigation and quick iteration by moving the LLM call into the playground.
- For transitioning from prototyping to production, 'weave' evaluations are used before deploying an agent.
8. ✅ Verification and Quality Assurance Processes
- Evaluate each task individually to ensure it performs as required and contributes effectively to the overall workflow.
- Use three separate scorers to evaluate 'generate searches' tasks: search value score, search recency score, and search count.
- Search value score evaluates the value and topical relevance of queries using a Large Language Model (LLM).
- Search recency score assesses temporal relevance using an LLM, emphasizing the importance of recency and time sensitivity in financial research.
- Search count provides a simple tally of generated queries.
- Utilize three separate models for 'generate searches': GPT4, mini GBT40, and 03 mini.
- Results are reviewed on the Weave evaluations page, allowing for comparison of evaluation outcomes.
9. 🔎 Deep Dive into Evaluation Metrics
- The interface enables comparison and differentiation of every option, facilitating iteration, optimization, and critical deployment decisions.
- Organized data presentation allows for efficient data-driven decision-making processes.
- The agentic workflow complements the evaluation process, enhancing overall decision-making efficiency.
10. 🔍 Role of Verification Agents in Ensuring Quality
- Verification agents act as meticulous auditors to ensure reports are internally consistent, clearly sourced, and free of unsupported claims.
- The presence of verification agents adds a layer of protection against delivering problematic results to analysts.
- Verification agents provide a pass or fail score for reports, offering feedback for improvement even on passing reports.
- A failing score from a verification agent triggers instructions to either discard the report and start anew or seek administrator assistance for resolution.
- Verification pass/fail scores can be recorded in the Weave system for data filtering, further analysis, or inclusion in future data sets for evaluation or training.
11. 🌟 Continuous Agent Improvement with Weave
- Weave is integral to the developer platform, streamlining the agent development workflow by continuously refining, optimizing, and adding new features. This ensures ongoing application improvements and a superior end-user experience.
- It allows developers to easily measure the impact and performance of enhancements, fostering data-driven updates and informed decision-making.
- Developers are encouraged to sign up for Weave to build and deploy AI agents and applications with confidence, leveraging its robust measurement capabilities.
- For example, using Weave, a development team reduced their AI model's error rate by 15% in just two months through iterative improvements informed by performance metrics.
