Medical AI models trained on Western datasets often fail in India's unique healthcare context, missing crucial aspects like linguistic diversity, local disease patterns, and healthcare system dynamics. To ensure we can leverage AI models safely in India, robust evaluation frameworks are critical.
KARMA (Knowledge Assessment and Reasoning for Medical Applications) is an open-source Python framework designed to evaluate medical AI systems with specific consideration for Indian healthcare realities. The framework provides specialized capabilities for multi-modal medical data including text, images, and audio, while implementing evaluation metrics that account for India-specific requirements. It includes 21 integrated medical datasets.
KARMA is built with extensibility as a core principle, through the registry decorators, developers can contribute new datasets/models the library with a single file addition. And through the CLI, evaluating models on multiple datasets is easy.