Support overriding the config.yaml with environment variables at runtime (e.g. in Docker)

[dbsanfte]

• Added support for environment-variable-driven configuration, introducing a script to dynamically process the configuration file at runtime.

• Updated the Dockerfile to install additional utilities (bash, curl, wget, and yq) needed by the script

• A more detailed config.yaml.example. This includes documentation for variable mapping, value types, and examples for easier customization.

[Copilot]

Copilot wasn't able to review any files in this pull request.

Files not reviewed (3)

• Dockerfile: Language not supported
• config.yaml.example: Language not supported
• process_config.sh: Language not supported

[dbsanfte]

Running the script like so:

dbsanfte@local$ MANIFOLD__COMPLETIONS__DEFAULT_HOST="asdf" MANIFOLD__MCPSERVERS__MANIFOLD__COMMAND="1234" ./process_config.sh

Building YAML path dictionary...
Found paths in YAML:
  embeddings.embed_prefix -> embeddings.embed_prefix
  embeddings.search_prefix -> embeddings.search_prefix
  reranker.host -> reranker.host
  reranker -> reranker
  google_gemini_key -> google_gemini_key
  embeddings.api_key -> embeddings.api_key
  mcpservers.manifold.args -> mcpServers.manifold.args
  single_node_instance -> single_node_instance
  database -> database
  port -> port
  embeddings.dimensions -> embeddings.dimensions
  database.connection_string -> database.connection_string
  mcpservers -> mcpServers
  data_path -> data_path
  mcpservers.manifold -> mcpServers.manifold
  completions -> completions
  mcpservers.manifold.env -> mcpServers.manifold.env
  embeddings.host -> embeddings.host
  embeddings -> embeddings
  hf_token -> hf_token
  completions.completions_model -> completions.completions_model
  anthropic_key -> anthropic_key
  mcpservers.manifold.command -> mcpServers.manifold.command
  host -> host
  completions.api_key -> completions.api_key
  completions.default_host -> completions.default_host
Direct match found: mcpservers.manifold.command -> mcpServers.manifold.command
Setting mcpServers.manifold.command = 1234
Direct match found: completions.default_host -> completions.default_host
Setting completions.default_host = asdf
Config file processed successfully!

Yields:

...

# Handles generation of text completions based on input prompts
completions:
  # OpenAI-compatible API endpoint
  default_host: "asdf"
  # Model identifier to use for completions
  completions_model: 'gpt-4o'
  # API key for the completions service (if required)
  api_key: ""

...

# MCP (Model Control Protocol) servers provide Manifold with data access
mcpServers:
  # Manifold's built-in MCP server
  manifold:
    command: 1234
    args: []
    env: {}

[Art9681]

Will be reviewing this soon. I apologize for the delay.

[Art9681]

@dbsanfte This looks good. A couple of thoughts:

• This won't work on MacOS since it requires newer version of bash. That's ok since in MacOS running a container is not ideal (no GPU passthrough).
• Are you able to test this on a Windows machine? I ask because i'm thinking we should just renamed the Dockerfile to something like linux.Dockerfile or whatever convention is best practice when we have multiple Dockerfiles for different OS, etc.
• If you have tested this in Linux (or Windows), rename it to windows.Dockerfile or linux.Dockerfile or whatever convention would be better if you have thoughts on that.

Let me know if you are able to test in both Windows and Linux and if not, I will provision a system and do it on whichever OS you cant. I'm open to your suggestions on my previous thoughts.

The new config.yaml.example is great. I really like how nice it looks now. Well done.

If we confirm it works in both Linux and Windows then i'm fine proceeding with the changes as-is.

[Art9681]

Tested in Linux environment and validated it works as intended.