Your Readme Is Unclear about how to do custom experiments

[Tylersuard]

You say it should be easy, it is not.

"experiment.py -- This is a single file where the 'meat' of the content is. It takes in an argument for out_dir, which is where it should create the folder and save the relevant information from the run."

What is the meat of the content? That does not make any sense to and English speaker. Could you please be more specific about what this file is, what it does, and how to modify it?

"plot.py -- This should take in the information from the run folders and create plots. The code should be clear and easy to edit."

What is a run folder?

"prompt.json -- Put information about your template here.
seed_ideas.json -- Put example ideas here. You can also try to generate ideas without any examples, and then pick the best one or two to put here."

How do I generate ideas without any examples? Should I stop the experiment in the middle, pick the ideas, then retry with the ideas in here, or what

"latex/template.tex -- We recommend using our latex folder, but be sure to replace the pre-loaded citations with ones that you would expect to be more relevant"

I have no idea what this means. Do I need to put citations in before I run the experiment?

You might say "this is all in the paper", but the paper is 186 pages long.

[callor]

The seed_ideas.json file requires information about the topic of the paper you want to create and how you will study it.
You need to do some pre-learning by running the following code:

cd templates/nanoGPT && python experiment.py --out_dir run_0 && python plot.py

Generate ideas by running the following code
Then we create an idea.json file and iterate over it to rewrite the code in the experiment.py file.

python launch_scientist.py --model "gpt-4o-2024-05-13" --experiment nanoGPT --num-ideas 2

[CaptainBobby1117]

The seed_ideas.json file requires information about the topic of the paper you want to create and how you will study it. You need to do some pre-learning by running the following code:

cd templates/nanoGPT && python experiment.py --out_dir run_0 && python plot.py

Generate ideas by running the following code Then we create an idea.json file and iterate over it to rewrite the code in the experiment.py file.

python launch_scientist.py --model "gpt-4o-2024-05-13" --experiment nanoGPT --num-ideas 2

yes I know but what about the experiment.py and plot.py, is the format a must but how to custom these files if my code is complex

[jli113]

same question, and in my case, I don't need experiment.py or polt.py

[conglu1997]

Please refer to this example from the community: #141 :)

Many more in the forks of this repo. I encourage you to take the time to read our paper as many of your questions will already have been answered.

The crucial things are to match the output JSON schema, everything else is customizable. Non-standard formats should be doable with a bit of hacking.

[conglu1997]

You can now check all 3 new community templates to see what others have been building! :)