2025-09-13 –, Track 1
Prompt engineering has become a core skill in working with LLMs, yet writing effective prompts is tedious, inconsistent, and often requires multiple attempts. Most prompt development today is done manually, relying on trial and error, which doesn’t scale well, especially in production environments where consistency is crucial. This talk presents a practical approach to optimizing prompts by treating them as evolving tools. The process includes refining prompts based on feedback and ensuring the outputs meet structure and quality requirements, with the help of Pydantic to validate and enforce output standards. This helps create stable, production-ready prompts through continuous and iterative improvement.
Crafting a good prompt shouldn’t feel like guesswork. In this talk, we’ll introduce a method for improving prompts using feedback from humans or models. By evaluating prompts against task-specific datasets, scoring output quality, and refining prompts based on feedback, this method ensures continuous improvement and higher-quality results. We also use Pydantic to ensure the output adheres to the required structure and meets quality standards, making the process more reliable and aligned with production requirements.
Outline
- Challenges in prompt engineering
- Feedback-driven refinement
- Scoring prompts
- Rephrasing with feedback
- Output validation (Pydantic)
- Iterative improvement
Target Audience: ML engineers, data scientists, developers working with LLMs in production, and anyone looking to learn how to build robust AI workflows using open source tools.
- Familiarity with Python
- Basic understanding of LLMs
Beginner
Mahima Arora is a Senior Data Scientist on the Data & AI team at Red Hat, specializing in Generative AI applications. She develops AI-powered solutions that enhance efficiency and effectiveness, leading initiatives to optimize AI systems for greater impact. An open-source enthusiast, Mahima continuously explores new tools and technologies to expand her expertise and stay at the forefront of innovation.
Aarti Jha is a Senior Data Scientist at Red Hat, where she develops AI-driven solutions to streamline internal processes and reduce operational costs. She brings over 6.5 years of experience in building and deploying data science and machine learning solutions across industry domains.