feat: DIA-1402: V1-Submit Prompt auto-refinement job #214
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #214 +/- ##
==========================================
- Coverage 67.09% 66.94% -0.15%
==========================================
Files 44 45 +1
Lines 2103 2272 +169
==========================================
+ Hits 1411 1521 +110
- Misses 692 751 +59 ☔ View full report in Codecov by Sentry. |
server/prompt_improvement_skill.py
Outdated
| prompt_improvement_skill = TextGenerationSkill( | ||
| name="prompt_improvement", | ||
| instructions="Improve the user prompt for the provided LLM model to complete the task using the provided input variables, with the provided user prompt as a starting point. Variables can be accessed in the user prompt using the format {variable_name} (only the variable values are used, not their names). Make sure your prompt produces output that will continue to conform to the provided json schema. Provide your reasoning for the changes you made to the prompt.", | ||
| input_template=''' |
There was a problem hiding this comment.
Let's move it to a separarate file, so we could keep refining it easily.
A couple of a prompt eng best practices that we could incorporate right away:
- split each variable in a well defined block, for example:
Model: {model}-->## Model\n{model}\n\n - add fewshot examples
- add
# Instructions:or# Follow the steps:section with a numbered list of instructions - add
# Things to avoid:section where we patch the mistakes
prompt engineering for refinement is out of the scope for sure, but moving input_template in a separate file and leaving instructions empty would allow us to easily iterate on that in the future updates
There was a problem hiding this comment.
I want to keep all the deps for the PromptImprovementSkill together, not sure putting just the input template in a separate file would be any cleaner.
But one thing I want to get your opinion on re prompt engineering is the trick of factoring out all the prompt contents that don't contain variables into the system prompt, and leaving the variables in the user prompt. This way we can easily use prompt caching on the system prompt, and it would also change the performance in some way - not sure if positive or negative, but I suspect positive, and I wanted to trial it here first, and if it works here bring it to LSE. wdyt?
There was a problem hiding this comment.
Updated the prompt. Keeping as separate system and user prompts for now, lmk if you have a strong opinion about that
- make agent with a teacher runtime serializable - add test
Add a prompt improvement skill and an endpoint to call it through the server
TODO
[x] validation
[x] test coverage
[x] clean up our data models - started in this new codepath, can extend to existing codepaths later