2025-09-15 –, Track 1
Medical AI models trained on Western datasets often fail in India's unique healthcare context, missing crucial aspects like linguistic diversity, local disease patterns, and healthcare system dynamics. To ensure we can leverage AI models safely in India, robust evaluation frameworks are critical.
KARMA (Knowledge Assessment and Reasoning for Medical Applications) is an open-source Python framework designed to evaluate medical AI systems with specific consideration for Indian healthcare realities. The framework provides specialized capabilities for multi-modal medical data including text, images, and audio, while implementing evaluation metrics that account for India-specific requirements. It includes 21 integrated medical datasets.
KARMA is built with extensibility as a core principle, through the registry decorators, developers can contribute new datasets/models the library with a single file addition. And through the CLI, evaluating models on multiple datasets is easy.
The Indian Medical AI Challenge
Demonstrating how Western-trained models fail on Indian medical data, from code-switching in consultations to local drug names and terminology. Real examples from clinical practice showing evaluation gaps.
Framework Architecture & Design
KARMA's Pythonic design using decorator-based registry system for adding new models and datasets, intelligent caching through duckdb for expensive model outputs, and clean separation of concerns. Model-agnostic interface supporting various backends from Hugging Face to AWS Bedrock.
Novel Evaluation Metrics
Introduction of Semantic WER for handling medical code-switching and transliteration variations. Keyword WER for ensuring critical medical terms are captured accurately. Implementation details and performance comparisons.
Live Demo & Code Examples
Installation in one command, evaluation in three lines of code. Adding custom models and datasets with single file additions. Benchmark results comparing SOTA models on Indian medical data.
Community & Extensions
How researchers can contribute new datasets, models, and metrics. Current roadmap for Indian language support. Integration with existing ML workflows and evaluation pipelines.
uv
Additional Resources – Target Audience –Intermediate
Lead Data Scientist currently working with Eka.care, I have 7 years of experience building ML systems.
Talk to me about all things LLM Evals, MCP, and Retrieval!