Parallelize router initialization

[noobHappylife]

Title

Use ProcessPoolExecutor to parallelize set_model_list in Router initialization

Relevant issues

Related to #7647

Type

🆕 New Feature

Changes

Use ProcessPoolExecutor to parallelize set_model_list in Router initialization

[REQUIRED]

A test script to test the router creation time

import time
from litellm import Router

# Function to create a list of model configurations
def generate_model_configs(num_models):
    model_configs = []
    for i in range(num_models):
        model_config = {
            "model_name": f"model_{i}",
            "litellm_params": {
                "model": f"openai/model_{i}",
                "api_key": "your-api-key",
                "api_base": "https://api.openai.com/v1"
            }
        }
        model_configs.append(model_config)
    return model_configs

# Function to test router creation time
def test_router_creation(num_models):
    model_configs = generate_model_configs(num_models)

    start_time = time.time()
    router = Router(model_list=model_configs)
    end_time = time.time()

    creation_time = end_time - start_time
    print(f"Router num models: {len(router.model_list)}")
    print(f"Time to create router with {num_models} models: {creation_time:.4f} seconds")

# Test with varying number of models
model_counts = [10, 20, 30, 40, 50]
for count in model_counts:
    test_router_creation(count)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 11, 2025 7:10am

[krrishdholakia]

Hey @noobHappylife this is a great PR. Can you add some testing for your change.

Possibly in this file -

litellm/tests/local_testing/test_router_client_init.py

Line 26 in 8e2653c

async def test_router_init():

we typically add a unit test for all functions in router.py

[noobHappylife]

Hey @noobHappylife this is a great PR. Can you add some testing for your change.

Possibly in this file -

litellm/tests/local_testing/test_router_client_init.py

Line 26 in 8e2653c

async def test_router_init():

we typically add a unit test for all functions in router.py

Apology, would you suggest what kind of unit test to be included, since this change affects the initiation step, and the checks has passed (I assumed includes the test_router_init). As this change are not supposed to alter any of current behavior (regarding input/output).

[krrishdholakia]

@noobHappylife add a test for your helper utils to just confirm they work as expected.

this ensures any future changes/refactor don't introduce regressions