Added Tavily search functionality #80
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
WalkthroughThis pull request introduces web scraping capabilities to the system by adding support for the Tavily API. The changes include adding a new environment variable for the Tavily API Key, creating a Changes
Sequence DiagramsequenceDiagram
participant User
participant Orchestrator
participant WebSearcher
participant TavilyAPI
User->>Orchestrator: Sends search query
Orchestrator->>WebSearcher: Invoke web search
WebSearcher->>TavilyAPI: Request web search
TavilyAPI-->>WebSearcher: Return search results
WebSearcher->>Orchestrator: Process and return results
Orchestrator->>User: Provide search results
Possibly related PRs
Poem
✨ Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
examples/example-api.ts (3)
109-149: Consider reducing verbose debug output or gating logs behind a log level.
While the debug logs are helpful for development, consider either removing them in production or routing them through a logging framework. Also, ensureprocess.env.TAVILY_API_KEY!is safely validated (the env schema helps, but handle any edge cases gracefully).- console.log("DEBUG: Full web_searcher context:", { - payload, - type: typeof payload, - keys: payload ? Object.keys(payload) : "no keys", - }); + this.logger.debug("web_searcher context", { + payload, + type: typeof payload, + keys: payload ? Object.keys(payload) : "no keys", + });
237-240: Replace development logs with a structured logging approach or remove them.
These “DEBUG” logs may clutter production logs, so consider gating them behind log-level checks.
283-284: Use consistent debug logging strategy.
As with other debug statements, unify or remove for clarity.packages/core/src/core/io/scraper.ts (1)
1-123: Overall implementation is good; consider removing unused function and parameterizingmaxResults.
cleanContentmethod is declared but not referenced. If it’s no longer needed, remove it.maxResults: 5is hard-coded. Exposing it as a parameter could enhance flexibility.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (8)
.env.example(1 hunks)examples/example-api.ts(7 hunks)packages/core/package.json(1 hunks)packages/core/src/core/env.ts(1 hunks)packages/core/src/core/io/scraper.ts(1 hunks)packages/core/src/core/orchestrator.ts(8 hunks)packages/core/src/core/processors/master-processor.ts(2 hunks)packages/core/src/core/utils.ts(2 hunks)
🔇 Additional comments (15)
examples/example-api.ts (5)
2-9: Documentation changes look good.
Thanks for clarifying the focus of this example.
27-30: Imports for WebSearcher and searchedContentSchema look fine.
These additions properly align with the new Tavily API integration.
35-35: ChromaVectorDB name updated to "api_agent".
This change looks consistent with the new context.
304-307: Intro messages look good.
They clearly inform users about the available features.
317-321: Graceful removal of IO handlers is appropriate.
Shutdown logic appears coherent and properly finalizes your orchestrator state.packages/core/src/core/env.ts (1)
18-18: New required TAVILY_API_KEY environment variable is properly declared.
This ensures the key is always present at runtime.packages/core/src/core/processors/master-processor.ts (2)
99-101: LGTM! Clear guidelines for web search handling.The separation between immediate and scheduled web searches using
suggestedOutputsandupdateTasksrespectively provides a clear pattern for handling different search scenarios.
115-118: LGTM! Improved output suggestion flow.The modified flow that prioritizes user feedback before action handlers will provide better user experience and more responsive interactions.
packages/core/src/core/utils.ts (4)
178-186: LGTM! Enhanced debug logging for raw LLM responses.Added logging of raw responses will help in debugging LLM parsing issues.
191-198: LGTM! Added successful parse logging.Logging successful JSON parsing provides better visibility into the processing flow.
206-216: LGTM! Improved error context in parse failures.Enhanced error logging with position and surrounding context will make debugging parse failures much easier.
231-231: LGTM! Added schema logging for validation failures.Including the schema in validation error logs will help identify schema mismatches.
packages/core/src/core/orchestrator.ts (1)
450-450: LGTM! Added fallback data for action dispatch.The addition of fallback data
{ query: processed.content }when output.data is not provided ensures the action handler always receives valid input..env.example (1)
30-31: LGTM! Well-documented environment variable.The addition of TAVILY_API_KEY with a clear comment and URL reference follows good documentation practices.
packages/core/package.json (1)
37-37: Verify stability of @tavily/core package.The package version (0.0.2) is very early in development which could indicate an unstable API, potential breaking changes, and limited production readiness.
Let's check the package's current state:
Consider:
- Monitoring for updates and breaking changes
- Testing thoroughly before deploying to production
- Having a rollback plan in case of API instability
✅ Verification successful
Early version (0.0.2) is an official beta release.
This is the official Tavily JavaScript SDK, currently in beta. While breaking changes are possible, the package is actively maintained with comprehensive documentation and support channels available at [email protected].
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify @tavily/core package status # Check latest version and release frequency echo "Latest version and publish dates:" npm view @tavily/core time --json | jq # Check download statistics for popularity/adoption echo -e "\nDownload statistics:" npm view @tavily/core downloads --json | jq # Check for open issues and PRs to assess maintenance echo -e "\nGitHub activity:" gh api -X GET search/repositories\?q\=repo:tavily/core | jq '.items[0] | {stars: .stargazers_count, forks: .forks_count, open_issues: .open_issues_count}'Length of output: 683
Script:
#!/bin/bash # Description: Get more package details and verify official status # Get package metadata echo "Package details:" npm view @tavily/core --json | jq '{ description, homepage, repository, license, author, maintainers }' # Check documentation echo -e "\nREADME content:" npm view @tavily/core readmeLength of output: 5651
| @@ -282,6 +282,8 @@ export class Orchestrator { | |||
| queue.push({ data: initialData, source: sourceName }); | |||
| } | |||
|
|
|||
| console.log("DEBUG: Queue ============= : ", queue); | |||
There was a problem hiding this comment.
🛠️ Refactor suggestion
Replace console.log with logger.debug.
Multiple console.log statements have been added for debugging. These should use the existing logger.debug for consistency and better control over log levels.
Apply this pattern for all debug logs:
-console.log("DEBUG: Queue ============= : ", queue);
+this.logger.debug("Orchestrator.runAutonomousFlow", "Processing queue", { queue });Also applies to: 358-361, 397-397, 413-413, 439-446, 514-514, 537-537, 614-617
Adds a web_searcher IOHandler and a new scraper class. Allows example-api to search the web using Tavily API.
Bit messy right now, as I've left in the debugging console statements.
Summary by CodeRabbit
Release Notes
New Features
Dependencies
@tavily/corelibrary for web scraping supportConfiguration
Documentation
The release focuses on expanding API interaction and web search capabilities, with improved logging and error handling across the platform.