Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] CrewAI import during 23 seconds #1570

Open
Rhuax opened this issue Nov 8, 2024 · 0 comments
Open

[BUG] CrewAI import during 23 seconds #1570

Rhuax opened this issue Nov 8, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@Rhuax
Copy link

Rhuax commented Nov 8, 2024

Description

I'm using the latest CrewAI version, and I'm noticing very low speed when importing the library.
I created a profiling code (thx gpt4) and CrewAI takes around 23 seconds to only import.

crewai: 23.593604 seconds
crewai.agent: 22.445331 seconds
crewai.llm: 22.024928 seconds
google.cloud.aiplatform.compat: 1.797949 seconds
crewai.flow.flow: 1.071128 seconds
crewai.flow.flow_visualizer: 1.069470 seconds
pyvis.network: 1.068562 seconds
litellm.llms.anthropic.cost_calculation: 1.008161 seconds
litellm.utils: 1.007175 seconds

I'm attaching the code used, genai_core is my custom library that actually imports CrewAI.

import time
import importlib
import sys
from contextlib import contextmanager

class ImportProfiler:
    def __init__(self):
        self.import_times = {}

    def log_import_time(self, name, start_time, end_time):
        elapsed_time = end_time - start_time
        if name not in self.import_times:
            self.import_times[name] = elapsed_time

    def report(self):
        sorted_imports = sorted(self.import_times.items(), key=lambda item: item[1], reverse=True)
        
        print("\nTop 10 Slowest First-Level Imports:")
        for mod, duration in sorted_imports[:10]:
            print(f"{mod}: {duration:.6f} seconds")

@contextmanager
def track_imports(profiler, main_lib):
    original_import = __import__

    def custom_import(name, globals=None, locals=None, fromlist=None, level=0):
        fromlist = fromlist or []  # Handle None fromlist
        if name.startswith(main_lib) or any(f.startswith(main_lib) for f in fromlist):
            start_time = time.time()
            module = original_import(name, globals, locals, fromlist, level)
            end_time = time.time()
            profiler.log_import_time(name, start_time, end_time)
            return module
        elif level == 0:
            start_time = time.time()
            module = original_import(name, globals, locals, fromlist, level)
            end_time = time.time()
            profiler.log_import_time(name, start_time, end_time)
            return module
        return original_import(name, globals, locals, fromlist, level)

    builtins_import_backup = sys.modules['builtins'].__import__
    sys.modules['builtins'].__import__ = custom_import
    try:
        yield
    finally:
        sys.modules['builtins'].__import__ = builtins_import_backup

if __name__ == "__main__":
    profiler = ImportProfiler()
    library_to_import = 'genai_core'  # Replace with the library you want to profile

    with track_imports(profiler, library_to_import):
        try:
            importlib.import_module(library_to_import)
        except ImportError as e:
            print(f"Failed to import {library_to_import}: {e}")

        # Example for from-import statement
        # This simulates: `from your_main_library import module1`
        try:
            module1 = importlib.import_module(f"{library_to_import}.cores")
        except ImportError as e:
            print(f"Failed to import from {library_to_import}: {e}")

    profiler.report()

Steps to Reproduce

Just import it

Expected behavior

.

Screenshots/Code snippets

.

Operating System

Ubuntu 20.04

Python Version

3.10

crewAI Version

0.76.9

crewAI Tools Version

0.13.4

Virtual Environment

Venv

Evidence

.

Possible Solution

.

Additional context

.

@Rhuax Rhuax added the bug Something isn't working label Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant