Microsoft Research - Keynote: Multimodal Generative AI for Precision Health | Microsoft Research Forum
Hoifung Poon, General Manager at Microsoft Health Futures, discusses the potential of generative AI in precision healthcare. The main challenge in biomedicine is the low response rate to treatments like immunotherapy, which only works for 20-30% of patients. AI can help by analyzing population-scale real-world data from digitized patient records, transforming each patient journey into a mini-trial. This approach can improve drug development efficiency and patient care by predicting medical events and understanding treatment responses. Poon highlights the development of GigaPath, a digital pathology model that scales AI to whole-slide images, and BiomedParse, a model for multimodal analysis, both of which have shown promising results. The ultimate goal is to democratize high-quality healthcare and reduce costs by leveraging AI to simulate clinical trials and improve treatment matching.
Key Points:
- Generative AI can transform patient data into actionable insights, improving treatment response rates.
- AI models like GigaPath and BiomedParse enhance analysis of medical images and multimodal data.
- Using AI, healthcare systems can simulate clinical trials, reducing costs and improving accessibility.
- AI-driven insights can help identify why certain patients do not respond to treatments like Keytruda.
- Collaboration with health systems and academia is crucial for advancing AI in healthcare.
Details:
1. 🎤 Introduction to Healthcare AI
1.1. Microsoft's Role in Healthcare AI
1.2. Collaborations and Clinical Impact
2. 🔬 Challenges in Cancer Treatment
- A significant challenge in cancer treatment is that many patients do not respond to prescribed treatments, indicating a critical issue in biomedicine.
- Immunotherapy, although an advanced treatment option, shows overall response rates of only 20 to 30% in cancer patients, underscoring the need for more effective therapies.
- Clinical trials represent a crucial option for patients who have exhausted standard treatments, yet only a small portion of patients in the US find matching trials, pointing to a lack of accessibility and resource allocation.
- The gap in clinical trial access highlights the necessity for improving infrastructure and resources to facilitate better patient-trial matching processes.
3. 💡 AI's Role in Drug Development
- Cancer trials often fail due to lack of patients, highlighting the need for more efficient recruitment strategies.
- Drug development is costly, requiring billions in investment and over a decade to bring a new drug to market.
- Precision health necessitates the creation of more drugs tailored for smaller patient populations, increasing the complexity and cost of development.
- Early drug discovery accounts for only 10-20% of total drug development costs, indicating that the majority of expenses are incurred during later stages.
- The major costs arise from clinical trials and post-market activities, with phase-three cancer trials alone costing hundreds of millions.
- AI provides opportunities to leverage population-scale real-world evidence, potentially reducing the time and cost of drug development.
- The rapid digitization of patient records across healthcare systems offers a wealth of data for AI to analyze, facilitating more informed decision-making in drug development.
- Billions of data points collected through routine clinical care can be utilized by AI to enhance precision medicine and streamline clinical trials.
4. 🔍 Leveraging Real-World Data
- Patient journeys serve as individual trials offering new insights, providing population-scale benefits when analyzed collectively.
- Cancer patient journeys, consisting of de-identified clinical notes and other modalities like medical imaging and multi-omics, deliver comprehensive data.
- Integrating multiple modalities is crucial for forming a complete patient representation, overcoming the limitations of isolated data types.
- Precision health leverages machine learning to predict important medical events (e.g., disease progression, tumor response) through multimodal patient journeys.
- Patient journeys are longitudinal, often with missing, noisy, and biased data, posing predictive challenges.
- Gen-AI has potential to overcome precision health challenges by utilizing incomplete and complex data effectively.
- Incorporating real-world examples, such as specific cancer patient case studies, could enhance understanding of practical applications.
5. 🤖 Generative AI for Precision Health
- Generative AI enables the compression of all observable patient information into a patient embedding, aiding in the prediction of missing information and medical events.
- The use of population-scale, real-world data allows for high-fidelity patient embeddings that act as digital twins, facilitating patient reasoning at a large scale.
- After a cancer diagnosis, generative AI can provide millions of opinions from similar patients almost instantaneously, reducing the time and resources needed for second opinions.
- This technology enables the interrogation of treatment pathways and longitudinal outcomes, improving patient care immediately.
- It allows for the comparison of non-responders and exceptional responders to treatments, such as understanding why 80% of patients do not respond to Keytruda.
- Generative AI helps unlock new capabilities from population-scale real-world evidence, challenging and advancing current healthcare practices.
6. 🧠 Innovations in Digital Pathology
- The field of digital pathology faces a significant competency gap in frontier models for non-text modalities in biomedicine, indicating a need for specialized training and development.
- A general recipe for self-supervision involves pre-training modality-specific encoders and decoders to effectively compress and decompress data, which is vital for biomedicine applications.
- Understanding tumor microenvironments through digital pathology is crucial for addressing immunotherapy resistance, highlighting its importance in cancer treatment.
- Pathology slides are extremely large, leading to an exponential increase in computational requirements due to the quadratic growth in transformer models.
- Dilated attention, an approach adapted from speech recognition, aids in overcoming computational challenges by selecting representatives for message passing in larger data blocks.
- GigaPath, a groundbreaking development in collaboration with Providence Health System and University of Washington, represents the first digital pathology foundation model capable of scaling transformers to whole-slide images.
- The impact of GigaPath is evident in its publication in Nature and over half a million downloads worldwide, demonstrating significant adoption and influence in the field.
7. 🌐 Multimodal Integration in Biomedicine
- Progress in multimodal biomedicine includes advances in CT and spatial multi-omics.
- Unimodal pre-training is a foundational step, but integrating different modalities remains challenging.
- Each biomedical modality communicates distinct information, akin to different languages.
- A proposed solution is to use text as an interlingua to integrate these modalities, similar to language translation models using English.
- Existing powerful models for biomedical text can be leveraged to serve as the interlingua.
- Readily available text-modality pairs, like pathology slides and reports, can facilitate this integration.
- By using unimodal encoders and decoders, and training a lightweight adapter layer, modalities can be aligned into a unified semantic space.
- This approach allows modalities to communicate in a common language and leverages existing knowledge.