2025-09-13 –, Track 1
Prompt engineering has become a core skill in working with LLMs, yet writing effective prompts is tedious, inconsistent, and often requires multiple attempts. Most prompt development today is done manually, relying on trial and error, which doesn’t scale well, especially in production environments where consistency is crucial. This talk presents a practical approach to optimizing prompts by treating them as evolving tools. The process includes refining prompts based on feedback and ensuring the outputs meet structure and quality requirements, with the help of Pydantic to validate and enforce output standards. This helps create stable, production-ready prompts through continuous and iterative improvement.
Crafting a good prompt shouldn’t feel like guesswork. In this talk, we’ll introduce a method for improving prompts using feedback from humans or models. By evaluating prompts against task-specific datasets, scoring output quality, and refining prompts based on feedback, this method ensures continuous improvement and higher-quality results. We also use Pydantic to ensure the output adheres to the required structure and meets quality standards, making the process more reliable and aligned with production requirements.
Outline
- Challenges in prompt engineering
- Feedback-driven refinement
- Scoring prompts
- Rephrasing with feedback
- Output validation (Pydantic)
- Iterative improvement
Target Audience: ML engineers, data scientists, developers working with LLMs in production, and anyone looking to learn how to build robust AI workflows using open source tools.
- Familiarity with Python
- Basic understanding of LLMs
Beginner
Aarti Jha is currently a Senior Data Scientist at Red Hat, Bengaluru, India, where she develops AI-driven solutions to streamline processes and reduce operational costs for internal initiatives. With over six years of experience, she has previously led the development of search and recommendation systems for e-pharma at her prior organisation.
In her free time, Aarti enjoys bringing her creative visions to life through sketching.
Mahima Arora is a Senior Data Scientist with the Data & AI team at Red Hat, specializing in Generative AI applications. She focuses on developing AI-powered solutions to streamline internal operations and has led initiatives in optimizing AI systems for greater efficiency and impact. An open source enthusiast, she loves exploring tools and technologies to continuously expand her work knowledge and stay at the forefront of innovation.