When I first saw a demo of GPT-4V in late 2023, a radiologist colleague pointed at the screen and said: this is the first time I've seen an AI system look at an image and reason about it the way a clinician would. Not perfectly - there were real limitations. But the reasoning modality was unmistakably different from what had come before.
Multimodal AI is the capability that I believe will have the largest practical impact on enterprise AI products over the next three years. Not because the models are perfect, but because the vast majority of real-world information is not text. Medical images, manufacturing inspection cameras, retail product photos, video surveillance, audio recordings, PDFs with embedded charts - this is where the information actually lives. Until models could reason natively across these modalities, AI was only touching a fraction of the available signal.
What Multimodal AI Is (And Is Not)
A multimodal AI model takes multiple types of input - text, images, audio, video, documents - and reasons across them in a single pass. This is different from:
- Chained single-modal models: Where you run an image through a separate vision model to get a text description, then feed that text to a language model. This approach loses information at every handoff and cannot reason about relationships between modalities that require joint processing.
- Specialized vision models: Models trained only on images for specific tasks (object detection, image classification, OCR). These are excellent at their narrow task but cannot answer open-ended questions about what they see or integrate image understanding with broader reasoning.
A true multimodal model processes all inputs in a shared representational space - the same underlying attention mechanism operates across text tokens, image patches, and audio spectrograms. This enables reasoning like: given this chest X-ray, this patient history text, and these prior scan images, what has changed and what might explain it? That reasoning pattern is impossible without native multimodal integration.
How It Works: The Technical Basics
You do not need to understand this at the level of a researcher to use it effectively as a PM, but a basic mental model helps when you are scoping what a multimodal system can and cannot do.
Transformers as the Unifying Architecture
The transformer architecture - originally designed for text - turned out to be remarkably generalizable. The key insight is that almost any structured input can be converted into a sequence of tokens that a transformer can process:
- Text: Words or sub-words are tokenized directly.
- Images: Images are divided into patches (e.g., 16x16 pixel squares) and each patch is embedded into a vector. These image patch embeddings are processed alongside text token embeddings by the same transformer layers.
- Audio: Audio is converted into a spectrogram (a visual representation of frequency over time) and then treated similarly to images. Or processed using a dedicated audio encoder that outputs embeddings compatible with the language model.
- Video: Extended to the temporal dimension - sequences of image frames, potentially with synchronized audio, are processed as sequences of spatial-temporal token sequences.
The training objective - predict the next token - generalizes across modalities. Models trained on large corpora of interleaved text and images develop representations that encode relationships between what things look like and how they are described. This is what gives multimodal models their ability to reason in natural language about visual inputs.
Key Models You Should Know
- GPT-4o (OpenAI): Natively multimodal - text, images, and audio processed in a unified model. Strong on document understanding, image Q&A, and combined text-image reasoning. The o stands for omni - treating all modalities as first-class inputs.
- Gemini 1.5 Pro / 2.0 Pro (Google): Designed from the ground up as multimodal with a 1M token context window that handles video at full length. Strongest multimodal model for video understanding tasks.
- Claude 3.5 / 3.7 (Anthropic): Strong image and document understanding - particularly for PDFs, charts, tables, and mixed text-image documents. The best model I have worked with for reasoning about complex document layouts.
Applications Across Industries
Healthcare: Radiology and Medical Imaging
This is the multimodal application with the highest stakes and the most active clinical research. The workflow that AI is transforming:
- Report generation: Radiologists review images and dictate reports. Multimodal AI can draft the report from the image, patient history, and prior scan comparisons, leaving the radiologist to review and sign off rather than originate. Studies have shown 30-50% reduction in report turnaround time at institutions running AI-assisted radiology workflows.
- Abnormality detection: AI systems trained on millions of annotated scans can flag potential abnormalities for radiologist review, functioning as a second reader that catches what human readers miss under high volume and fatigue conditions.
- Pathology: Digital pathology slides are high-resolution images where multimodal AI can assist in grading, classification, and biomarker identification. Foundation models for pathology (CONCH, PLIP, UNI) trained on large pathology image corpora are demonstrating strong performance on downstream clinical tasks.
Retail: Visual Search and Product Intelligence
Visual search - finding a product by uploading an image of something you saw and want to buy - is one of the highest-conversion features in e-commerce. Pinterest Lens, Google Lens, and retailer-specific implementations drive meaningful revenue from customers who cannot describe what they want but can show it.
Beyond visual search, multimodal AI enables:
- Automated product tagging: Upload a product image, receive a complete set of attributes (color, material, style, occasion) for catalog search and recommendation. Reduces the manual work of catalog enrichment by 60-80%.
- Visual merchandising analysis: Computer vision systems that analyze store shelf images to identify out-of-stock positions, planogram compliance failures, and competitor share-of-shelf.
- Virtual try-on: Multimodal models that can render products onto customer images for apparel, glasses, and cosmetics with realistic physics simulation.
Manufacturing: Quality Inspection and Process Monitoring
Computer vision is transforming manufacturing quality inspection. What multimodal specifically adds is the ability to combine visual inspection with textual context:
- This part was manufactured on Line 3, Shift 2, with Tool ID 447. Given this image of the finished part, does the surface quality match the expected profile for these parameters?
- Integrating vision data from inspection cameras with sensor data from the manufacturing process to identify root causes, not just defects.
Document Understanding
Enterprise documents are rarely pure text. Contracts, financial reports, regulatory submissions, clinical trial protocols - these are PDFs with tables, charts, footnotes, headers, and complex layout structures that pure text extraction destroys.
Multimodal document understanding treats the document as an image (preserving layout) and as text (enabling search and extraction) simultaneously. Applications include:
- Contract intelligence: extract key terms from contracts while understanding tables and formatted schedules that text extraction garbles
- Financial document analysis: reason about charts, graphs, and tables in annual reports alongside the narrative text
- Regulatory submission review: process complete FDA submission packages including figures, appendices, and embedded data tables
What Product Managers Need to Know
Multimodal Does Not Mean Magic
The failure mode I see most often is PMs scoping multimodal AI products as if the model will figure out what to do with any image put in front of it. Models have limitations in image resolution, ability to read fine text in images, reasoning about spatial relationships, and handling of unusual image types. Scope carefully against the specific modality and task.
Evaluation Is Harder
Evaluating text outputs is hard. Evaluating multimodal outputs is harder. A model that correctly identifies the main finding in a chest X-ray 95% of the time may miss certain finding types at 15%. Building evaluation sets that cover the full distribution of inputs - including the long tail of unusual cases - requires clinical or domain expertise, not just engineering.
Data Privacy in Multimodal Contexts
Images often contain more sensitive information than text. A photo of a patient includes their face, potentially identifiable by facial recognition. A medical image may contain embedded patient identifiers (burned in from legacy PACS systems). Video of a factory floor may contain safety violations you did not intend to capture. Design your data handling architecture for multimodal inputs with the most sensitive modality in mind, not the least.
Multimodal AI does not change what AI can do. It dramatically expands the surface area of problems AI can address. That expansion is where the next generation of high-value AI products will be built.
If you are building enterprise AI products and have not yet evaluated multimodal capabilities for your use case, that evaluation should be on your roadmap for this quarter. The capability is real, the applications are broad, and the early movers in multimodal-native product design are accumulating data advantages that will compound over the next several years.
