Remove optional dependencies from Poetry resolver (#72)

* Isolate extra requirements

* Refactor workflows to use `poetry run` for installing extra dependencies.

.github/workflows/integration_test.yml

@@ -53,6 +53,9 @@ jobs:
     - name: Install dependencies
       run: poetry install --no-interaction --all-extras
 
+    - name: Install extra dependencies
+      run: poetry run pip install -r requirements-extra.txt
+
     - name: Run integration tests
       env:
         OPENAI_API_KEY: fake_key_1337

.github/workflows/integration_test_minimal.yml

@@ -53,6 +53,9 @@ jobs:
     - name: Install dependencies
       run: poetry install --no-interaction
 
+    - name: Install extra dependencies
+      run: poetry run pip install -r requirements-extra.txt
+
     - name: Run integration tests
       env:
         OPENAI_API_KEY: fake_key_1337

.github/workflows/test.yml

@@ -53,6 +53,9 @@ jobs:
     - name: Install dependencies
       run: poetry install --no-interaction --all-extras --with dev
 
+    - name: Install extra dependencies
+      run: poetry run pip install -r requirements-extra.txt
+
     - name: Run tests
       run: poetry run pytest
 

pyproject.toml

@@ -8,24 +8,14 @@ readme = "README.md"
 [tool.poetry.dependencies]
 python = "^3.10"
 langchain = "^0.2"
-crewai = { version = "^0.41", optional = true }
 setuptools = "^67.6.2"
-duckduckgo-search = "5.3.0b4"
-llama-index = { version = "^0.10.27", optional = true }
 langchain-experimental = "^0.0.62"
 python-dotenv = "^1.0.0"
-lunary = {version = "^1.1", optional = true}
 langchainhub = "^0.1.15"
 kuzu = "^0.4.2"
 cloudpickle = "^3.0.0"
 platformdirs = "^4.2.1"
 pydantic = "^2.7.1"
-# TODO: The following dependencies for caching package should be optional
-requests = "^2.31.0"
-curl-cffi = "^0.6.4"
-httpx = "^0.27.0"
-motleycache = "^0.0.4"
-pglast = {version = "^6.2", optional = true}
 langchain-openai = "^0.1.22"
 
 [tool.poetry.group.dev.dependencies]
@@ -43,12 +33,6 @@ ipykernel = "^6.29.4"
 nbsphinx-link = "^1.3.0"
 nbformat = "^5.10.4"
 
-[tool.poetry.extras]
-crewai = ["crewai"]
-llama-index = ["llama-index"]
-lunary = ["lunary"]
-pglast = ["pglast"]
-
 [build-system]
 requires = ["poetry-core"]
 build-backend = "poetry.core.masonry.api"

requirements-extra.txt

@@ -0,0 +1,6 @@
+motleycache
+lunary==1.1.4
+llama-index==0.11.2
+crewai==0.51.1
+duckduckgo-search==5.3.0b4
+pglast==6.3

tests/itest_golden_data/validating_agent_output_ipynb.json

@@ -1 +1 @@
-"### Comprehensive Analysis of AI Advancements in 2024\n\n#### Key Trends\n1. **Generative AI Integration**: Generative AI, which gained significant attention in previous years, is now becoming more useful for the general public. Tools like ChatGPT have reached mass adoption, and organizations are deriving substantial business value from these technologies.\n\n2. **Multimodal AI**: This trend involves AI systems that can process and integrate multiple types of data (e.g., text, images, audio) simultaneously. This capability enhances the versatility and applicability of AI across various domains.\n\n3. **Ethics and Safety**: There is a growing emphasis on the ethical deployment of AI and ensuring safety in AI applications. This includes addressing biases, ensuring transparency, and complying with evolving regulatory landscapes.\n\n4. **AI Democratization**: More people are experimenting with AI models, leading to a proliferation of small, specialized AI applications. This democratization is making AI accessible to non-tech individuals and small businesses.\n\n5. **Consolidation of Data and AI Leadership**: Organizations are streamlining their technology and data leadership roles, reducing the number of chief data and analytics officers to create more integrated and efficient leadership structures.\n\n#### Breakthrough Technologies\n1. **AI-Powered Scientific Discovery**: AI is being used to accelerate scientific research and discovery, particularly in fields like drug development and materials science.\n\n2. **Elastocalorics**: This emerging technology involves materials that can change temperature under mechanical stress, offering potential applications in energy-efficient cooling systems.\n\n3. **AI in Clean Energy**: AI systems are being developed to store clean energy as heat, which could significantly contribute to decarbonizing industries.\n\n4. **Advanced Large Language Models**: Continued advancements in large language models are enabling more sophisticated natural language processing capabilities, enhancing applications in customer service, content creation, and more.\n\n5. **Cutting-Edge Robotics**: Innovations in robotics, powered by AI, are leading to more autonomous and capable machines that can perform complex tasks in manufacturing, healthcare, and other sectors.\n\n#### Potential Industry Impacts\n1. **Manufacturing**: The manufacturing sector is expected to see the largest financial impact from AI, with significant improvements in efficiency, predictive maintenance, and supply chain optimization.\n\n2. **Healthcare**: AI is revolutionizing healthcare through advancements in diagnostics, personalized medicine, and robotic surgery, leading to better patient outcomes and reduced costs.\n\n3. **Finance**: The finance industry is leveraging AI for fraud detection, algorithmic trading, and personalized financial services, enhancing security and customer experience.\n\n4. **Retail**: AI is transforming retail by enabling personalized shopping experiences, optimizing inventory management, and improving customer service through chatbots and recommendation systems.\n\n5. **Energy**: AI is playing a crucial role in the energy sector by optimizing energy consumption, integrating renewable energy sources, and improving grid management.\n\n6. **Physics**: AI is being used to uncover mechanisms in physics, such as classifying phase transitions in materials and enhancing computational efficiency by coupling AI with fundamental physics. These applications are driving new paradigms in AI discovery and promoting advancements in theoretical and applied physics.\n\nIn conclusion, 2024 is poised to be a pivotal year for AI, with significant advancements and widespread adoption across various industries. The focus on ethical AI, multimodal capabilities, and breakthrough technologies will drive innovation and create substantial economic and societal impacts."
+"### Comprehensive Analysis of AI Advancements in 2024\n\n#### Key Trends in AI in 2024\n\n1. **Explosive Growth of Generative AI and Multimodal AI**:\n   - Generative AI continues to evolve, becoming more accessible and useful for non-technical users. This trend is marked by the proliferation of small, specialized AI models that individuals and businesses can tinker with.\n   - Multimodal AI, which integrates multiple types of data (e.g., text, images, audio), is gaining traction, enabling more sophisticated and versatile AI applications.\n\n2. **Quantum AI**:\n   - Quantum AI is emerging as a significant trend, promising to revolutionize computational capabilities and solve complex problems that are currently intractable with classical computing.\n\n3. **Explainable AI (XAI)**:\n   - There is a growing emphasis on making AI models more transparent and understandable. Explainable AI aims to demystify the \"black box\" nature of many AI systems, enhancing trust and accountability.\n\n4. **Edge AI**:\n   - The deployment of AI at the edge (i.e., on devices rather than centralized servers) is expanding. This trend is driven by the need for real-time processing and reduced latency in applications such as autonomous vehicles and IoT devices.\n\n5. **AI Governance**:\n   - As AI becomes more integrated into various aspects of society, the importance of AI governance is increasing. This includes developing frameworks for ethical AI use, data privacy, and regulatory compliance.\n\n6. **Intersection of AI and Sustainability**:\n   - AI is being leveraged to address environmental challenges, from optimizing energy use to monitoring climate change. This trend highlights the role of AI in promoting sustainability.\n\n#### Breakthrough Technologies in AI in 2024\n\n1. **Generative AI**:\n   - Generative AI remains a breakthrough technology, with significant investments from enterprises. It is being used to create content, design products, and even generate code, demonstrating its versatility and impact.\n\n2. **AI in Automation**:\n   - Automation technologies powered by AI are transforming industries by streamlining processes, reducing costs, and increasing efficiency. This includes robotic process automation (RPA) and AI-driven decision-making systems.\n\n3. **AI in Personalization**:\n   - AI technologies are enhancing personalization in various sectors, particularly in e-commerce and marketing. Advanced AI algorithms analyze user behavior to deliver tailored experiences and recommendations.\n\n4. **AI in Healthcare**:\n   - Breakthroughs in AI are revolutionizing healthcare, from diagnostic tools to personalized treatment plans. AI is improving patient outcomes and operational efficiency in medical facilities.\n\n5. **AI in Cybersecurity**:\n   - AI is playing a critical role in cybersecurity, helping to detect and respond to threats in real-time. Advanced AI models are being developed to identify vulnerabilities and protect against cyberattacks.\n\n6. **AI in Physics**:\n   - AI is making significant strides in the field of physics. Researchers have developed AI techniques to classify phase transitions in materials more efficiently than existing methods. AI is also being used to formulate physical theories by recognizing patterns in complex data sets, simplifying interactions in physical systems.\n\n#### Potential Industry Impacts of AI Advancements in 2024\n\n1. **Manufacturing**:\n   - The manufacturing industry is expected to see the largest financial impact from AI. AI-driven automation and predictive maintenance are enhancing productivity and reducing downtime.\n\n2. **Healthcare**:\n   - AI is transforming healthcare by enabling early diagnosis, personalized treatments, and efficient management of healthcare resources. This leads to improved patient care and reduced healthcare costs.\n\n3. **Finance**:\n   - In the finance sector, AI is being used for fraud detection, risk management, and personalized financial services. AI-driven analytics provide deeper insights and enhance decision-making.\n\n4. **Retail and E-commerce**:\n   - AI is revolutionizing retail and e-commerce by enhancing customer experiences through personalized recommendations, inventory management, and supply chain optimization.\n\n5. **Entertainment**:\n   - AI technologies, such as deepfake and generative AI, are being used in the entertainment industry to create realistic visual effects, de-age actors, and generate content, pushing the boundaries of creativity.\n\n6. **Energy**:\n   - AI is contributing to the energy sector by optimizing energy consumption, integrating renewable energy sources, and improving grid management, promoting sustainability and efficiency.\n\n---\n\nThis comprehensive analysis highlights the significant advancements in AI in 2024, the key trends shaping the industry, breakthrough technologies, and their potential impacts across various sectors, including the field of physics."

tests/test_agents/test_llms.py

@@ -1,11 +1,10 @@
 import pytest
-
 from langchain_openai import ChatOpenAI
 from llama_index.llms.openai import OpenAI
 
-from motleycrew.common.llms import init_llm
 from motleycrew.common import LLMFamily, LLMFramework
 from motleycrew.common.exceptions import LLMFamilyNotSupported
+from motleycrew.common.llms import init_llm
 
 
 @pytest.mark.parametrize(