Clean up cot_instructions placeholder

[wongjingping]

Changes

• Replaced cot_instructions with table_aliases. We prefer to leave cot_instructions as an experimental field.
• Users can insert table_aliases into their prompts more easily now by just directly sticking the placeholder {table_aliases} inside their prompt, without having to pass a combination of cli args + prompt changes + code changes.
• Rename join_hints to join_str to better match the variable name

Testing

We ran the following on gpu-inference

./run_model_cot.sh defog/sqlcoder-8b-dare-ties

Results:

Question File	Accuracy
instruct_basic_postgres.csv	92.50%
instruct_advanced_postgres.csv	81.25%
questions_gen_postgres.csv	83.81%

These are the same compared to when run on the main branch, and differences from our sota results are likely due to the different gpu hardware/setup.

[wendy-aw]

Thank you for cleaning it up!

[rishsriv]

To clarify the PR a bit, does this always lead to the creation of table aliases as a default - regardless of the CLI params that a user uses? :)

[wendy-aw]

Would we also need the table_alias field in generating the prompt in all other runners?

[rishsriv]

Would we also need the table_alias field in generating the prompt in all other runners?

Don't think we need to – performance for all generalist models decreased significantly after adding the table_alias field, so decided to intentionally not put it in!

[wongjingping]

To clarify the PR a bit, does this always lead to the creation of table aliases as a default - regardless of the CLI params that a user uses? :)

Yep that's right! The user can control that option directly within the prompt by just putting/omitting the placeholder

[wongjingping]

Would we also need the table_alias field in generating the prompt in all other runners?

Oh good point, I think we'll minimally need to add it to our api_runner.py for parity with our current setup. Just added it and the performance bumped back up:

Question File	Accuracy
instruct_basic_postgres.csv	97.50%
instruct_advanced_postgres.csv	84.38%
questions_gen_postgres.csv	88.10%

On a separate note, I think we can potentially refactor generate_prompt to just take in a pd.Series instead of explicitly listing everything out so we don't have to update all the runners each time we add a new prompt placeholder. Alternatively we can try to consolidate the common code in all the runners so we don't have to update all the runners everywhere.

[wendy-aw]

just take in a pd.Series instead of explicitly listing everything out so we don't have to update all the runners each time
Oh this is a good idea!
Anw the rest looks good to me!

[rishsriv]

Thanks for making the fix for the API runner!