feat(api): enhance `/search` endpoint with flexible filtering and dynamic aggregations #155

kouloumos · 2024-12-18T09:25:04Z

This PR is the first step towards addressing #154, enhancing the Bitcoin Search API with several new features and improvements:

Search API Enhancements:
- Support for filter operations (include/exclude).
- OR logic via array values.
- Dynamic aggregation fields and sub-aggregations.
- Switching between default and coredev indices.
For a deeper dive into the rationale and specifics of the new implementation, refer to Consolidate search functionality across products by enhancing Bitcoin Search API #154.
New /explore Page:
- Replaces the existing /sources page, generalizing document exploration to include sources, authors, and tags.
- Leverages the improved /search endpoint for a unified exploration experience.
- Provides threaded, flat, and summary view modes for all content types.
- Implements reusable data-fetching hooks for cleaner, modular code.
- Serves as a testbed for the enhanced /search API.
Search Page Enhancements:
- Support for dev mode document inspector for examining raw document data in search results.
  Set dev mode by running in the browser console:
```
localStorage.setItem('devMode', 'true')
```
Code Refactor:
- Includes a minor refactor originally proposed in refactor(api): use document ID instead of URL for content fetching #153, integrated here for consistency with related changes.

vercel · 2024-12-18T09:25:09Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
bitcoinsearch-app	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 3, 2025 10:04am

Emmanuel-Develops · 2025-01-27T23:41:53Z

src/utils/server/apiFunctions.ts

-    },
-  };
+const buildFilterQueryClause = (filter: Facet): QueryDslQueryContainer => {
+  if (Array.isArray(filter.value)) {


In what situation would you have the filter value as an array?

cc: @kouloumos

An array filter means OR clause between the filters in the array. For more you can read the comment on the line below this or the extended explanation in the connected issue.

Emmanuel-Develops

Works good! (Tested with postman)

Emmanuel-Develops · 2025-01-27T23:47:11Z

@kouloumos can you rebase on staging

kouloumos · 2025-01-28T09:35:52Z

can you rebase on staging

just did. I am not sure what happened and @0tuedon's d099438 commit made it here after the rebase with the latest staging. I'll just drop it.

Add support for filter operations (include/exclude), OR logic via array values, dynamic aggregation fields and sub-aggregations in the search API. Allow switching between default and coredev indices. This consolidates search functionality across products by making the Bitcoin Search API more flexible while maintaining backward compatibility.

kouloumos · 2025-01-28T11:36:29Z

I just updated the PR with a change in the Explore Page that takes advantage of the newly introduced API functionality for switching between indices to allow exploration of documents on both of our indices. ~~For this to work, we need to define a new COREDEV_INDEX env variable in our deployment, cc @Emmanuel-Develops~~ edit: no need for this after c4c1e7f

kouloumos · 2025-01-28T13:36:25Z

Just pushed an update that removes the need for index-related environment variables (INDEX and COREDEV_INDEX). Since index names aren't sensitive information, I moved them to a typed configuration in src/config/config.ts. This makes it easier to:

Add new indexes without requiring environment changes
Maintain type-safety across the application
Document available indexes directly in the code

Introduces a flexible explore page that generalizes source exploration to handle domains, authors, and tags. Consolidates API usage by leveraging the existing /search endpoint with aggregations instead of specialized source endpoints. The new implementation uses reusable hooks for data fetching and provides threaded, flat, and summary view modes across different content types.

Removes /sources and /sourceDocuments endpoints and their associated hooks now that their functionality has been consolidated into the /search endpoint used by the explore interface.

kouloumos · 2025-01-31T15:24:13Z

Just pushed a new commit that adds support for dev mode document inspector for examining raw document data in search results.

Set dev mode by running in the browser console:

localStorage.setItem('devMode', 'true')

The application was previously using URLs to fetch document content, but URLs are not guaranteed to be unique across documents. Switch to using Elasticsearch document IDs which are guaranteed unique identifiers. Update types, API endpoints, and components to pass document IDs instead of URLs throughout the content fetching flow.

Implement index selector dropdown to toggle between default and coredev indexes throughout the explore page. Update all data fetching hooks and components to support index selection, including document exploration, thread views, and content modal. This enables users to seamlessly switch between different elasticsearch indices while maintaining consistent data fetching across all views.

Moves index names from environment variables to config.ts since they aren't sensitive data. This provides type-safety and makes it easier to add new indexes. Updates API endpoints, components and documentation to use the new configuration.

Adds a clickable code icon to search result cards when dev mode is enabled (via localStorage) that opens a modal showing the raw document data.

Emmanuel-Develops · 2025-02-04T12:18:16Z

src/components/customResults/Result.tsx

-    </div>
+
+      {showModal && (
+        <DocumentModal


For devMode in the results page, is there a reason fetching the document again?

we can pull the data already and pass it along to render. The only difference is data exempted from the main query, i.e summary_embeddings will be unavaialable

it makes it easier:

it allows us to reuse the same component as is

I don't need to care about moving data

This is a mode that will not be used a lot

kouloumos requested a review from Emmanuel-Develops December 18, 2024 09:25

vercel bot had a problem deploying to Preview December 18, 2024 09:25 Failure

kouloumos force-pushed the api-improvements branch from 147de66 to d91b8b2 Compare December 18, 2024 09:29

vercel bot deployed to Preview December 18, 2024 09:30 View deployment

kouloumos force-pushed the api-improvements branch from d91b8b2 to 25d498f Compare December 18, 2024 09:33

vercel bot deployed to Preview December 18, 2024 09:34 View deployment

kouloumos force-pushed the api-improvements branch from 25d498f to bdd0bb1 Compare December 19, 2024 09:31

vercel bot deployed to Preview December 19, 2024 09:32 View deployment

kouloumos force-pushed the api-improvements branch from bdd0bb1 to 547cfb8 Compare December 20, 2024 08:50

vercel bot had a problem deploying to Preview December 20, 2024 08:50 Failure

kouloumos force-pushed the api-improvements branch from 547cfb8 to 2c4e9b8 Compare December 20, 2024 08:54

vercel bot deployed to Preview December 20, 2024 08:54 View deployment

vercel bot deployed to Preview December 20, 2024 10:47 View deployment

vercel bot deployed to Preview December 20, 2024 15:04 View deployment

kouloumos mentioned this pull request Dec 20, 2024

refactor(api): use document ID instead of URL for content fetching #153

Closed

Emmanuel-Develops force-pushed the staging branch 2 times, most recently from 27e6de5 to 2bcc1ec Compare December 31, 2024 15:46

Emmanuel-Develops mentioned this pull request Jan 2, 2025

Search page bitcointranscripts/registry#73

Merged

8 tasks

Emmanuel-Develops reviewed Jan 27, 2025

View reviewed changes

Emmanuel-Develops approved these changes Jan 27, 2025

View reviewed changes

kouloumos force-pushed the api-improvements branch from 0647ce3 to 7158263 Compare January 28, 2025 09:31

vercel bot deployed to Preview January 28, 2025 09:33 View deployment

kouloumos force-pushed the api-improvements branch from 7158263 to 9e642c9 Compare January 28, 2025 09:36

vercel bot deployed to Preview January 28, 2025 09:37 View deployment

vercel bot deployed to Preview January 28, 2025 11:33 View deployment

vercel bot deployed to Preview January 28, 2025 13:38 View deployment

kouloumos added 2 commits January 28, 2025 15:57

refactor: remove deprecated sources API endpoints and hooks

aa803aa

Removes /sources and /sourceDocuments endpoints and their associated hooks now that their functionality has been consolidated into the /search endpoint used by the explore interface.

kouloumos force-pushed the api-improvements branch from 9b54bf4 to c4c1e7f Compare January 28, 2025 13:59

vercel bot deployed to Preview January 28, 2025 13:59 View deployment

vercel bot deployed to Preview January 31, 2025 15:10 View deployment

kouloumos force-pushed the api-improvements branch from 8c9a812 to 2e5a701 Compare January 31, 2025 16:41

vercel bot deployed to Preview January 31, 2025 16:42 View deployment

kouloumos added 4 commits January 31, 2025 18:47

feat: add dev mode document inspector to result cards

846159e

Adds a clickable code icon to search result cards when dev mode is enabled (via localStorage) that opens a modal showing the raw document data.

kouloumos force-pushed the api-improvements branch from 2e5a701 to 846159e Compare February 3, 2025 10:03

vercel bot deployed to Preview February 3, 2025 10:04 View deployment

Emmanuel-Develops reviewed Feb 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(api): enhance `/search` endpoint with flexible filtering and dynamic aggregations #155

feat(api): enhance `/search` endpoint with flexible filtering and dynamic aggregations #155

kouloumos commented Dec 18, 2024 •

edited

Loading

vercel bot commented Dec 18, 2024 •

edited

Loading

Emmanuel-Develops Jan 27, 2025

Emmanuel-Develops Feb 4, 2025

kouloumos Feb 5, 2025

Emmanuel-Develops left a comment

Emmanuel-Develops commented Jan 27, 2025

kouloumos commented Jan 28, 2025

kouloumos commented Jan 28, 2025 •

edited

Loading

kouloumos commented Jan 28, 2025

kouloumos commented Jan 31, 2025

Emmanuel-Develops Feb 4, 2025

kouloumos Feb 5, 2025

feat(api): enhance /search endpoint with flexible filtering and dynamic aggregations #155

Are you sure you want to change the base?

feat(api): enhance /search endpoint with flexible filtering and dynamic aggregations #155

Conversation

kouloumos commented Dec 18, 2024 • edited Loading

vercel bot commented Dec 18, 2024 • edited Loading

Emmanuel-Develops Jan 27, 2025

Choose a reason for hiding this comment

Emmanuel-Develops Feb 4, 2025

Choose a reason for hiding this comment

kouloumos Feb 5, 2025

Choose a reason for hiding this comment

Emmanuel-Develops left a comment

Choose a reason for hiding this comment

Emmanuel-Develops commented Jan 27, 2025

kouloumos commented Jan 28, 2025

kouloumos commented Jan 28, 2025 • edited Loading

kouloumos commented Jan 28, 2025

kouloumos commented Jan 31, 2025

Emmanuel-Develops Feb 4, 2025

Choose a reason for hiding this comment

kouloumos Feb 5, 2025

Choose a reason for hiding this comment

feat(api): enhance `/search` endpoint with flexible filtering and dynamic aggregations #155

feat(api): enhance `/search` endpoint with flexible filtering and dynamic aggregations #155

kouloumos commented Dec 18, 2024 •

edited

Loading

vercel bot commented Dec 18, 2024 •

edited

Loading

kouloumos commented Jan 28, 2025 •

edited

Loading