[Bug]: Configuration behavior is unpredictable

[khushvind]

Is there an existing issue for the same bug?

• I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
• I have checked the existing issues.

Describe the bug

When I change the model, and api_key in config.toml, OpenDevin still runs with the previous model, and api_key specified in the last session. To use the new model and key, I also need to specify them again in the UI.

Specifying it once should be enough.

Current OpenDevin version

0.8.2

Installation and Configuration

make build; make run

Model and Agent

Model: claude-3-5-sonnet
Agent: CodeActAgent

Operating System

No response

Reproduction Steps

No response

Logs, Errors, Screenshots, and Additional Context

No response

[neubig]

We had a discussion on slack: https://opendevin.slack.com/archives/C06U8UTKSAD/p1722599155702189

Based on the discussion some of my personal suggestions for how we could improve the behavior now:

First, make consistent behavior of configuration:

1. Read from the defaults specified in config.py in the OpenDevin repo
2. Read from config.toml and override settings
3. (optional) Read from environment variables and/or command line and override settings

Here, 3. increases complexity, but it also could be handy because it allows overriding settings without writing to a file (e.g. for sensitive API keys).

Second, make UI-changed settings overwrite config.toml. This would allow the settings to be saved to and read from a consistent place.

[HenrikBach1]

Plz consider that your environment settings when using them in Dockerfile configuration use case, will be invalid as they are as of today in #3196.

[enyst]

First, make consistent behavior of configuration:

1. Read from the defaults specified in config.py in the OpenDevin repo
2. Read from config.toml and override settings
3. (optional) Read from environment variables and/or command line and override settings

Here, 3. increases complexity, but it also could be handy because it allows overriding settings without writing to a file (e.g. for sensitive API keys).

I think the current situation is, when running via:

• headless (main.py) - this works ✅
• cli.py - works ✅
• web UI - doesn't read any of it ❌
• resolver - not sure ❔
• external scripts - depends on scripts, evals explicitly do their own thing, except for custom llm configs, which they read from toml

Second, make UI-changed settings overwrite config.toml. This would allow the settings to be saved to and read from a consistent place.

Currently: (new from the time of this topic)

• UI saves settings in the filestore (locally, on local machine by default)
  • but it's a new format, a .json file
  • the toml file can be loaded from a particular location, but that feature is unused AFAIK
• has config handling in server module, not config module

Related:

• discussion and PR: Create endpoint for resetting settings #6840 (comment)
• PR: Refactor llm config from toml and clean up #6923