feat(#1): catalogue and recommender integration #37

xamcost · 2025-02-06T16:00:29Z

Description

Hi there! This PR integrates a catalogue instance, as well as a recommender instance, to the marketplace. More precisely, it implements:

catalogue browsing: when doing a search, the frontend actually makes a sparql query to retrieve offerings based on text matching with their title OR their description in the catalogue. The list of providers and the list of keywords corresponding to the offerings in the text matching query are also retrieved, for further filtering of the offering results. On the other hand, catalogue navigation is reworked to happen server-side, leveraging URL params.
recommendations upon browsing the catalogue, based on the query of the user, displayed alongside search results.
when the user selects an offering from the results, the offering details are fetched from the catalogue, as well as some similar offerings from the recommender.

How to test

You will need to port-forward to the Sedimark cluster to access the catalogue and recommender:

kubectl port-forward -n catalogue svc/jena-fuseki 3030:3030  # for the catalogue
kubectl port-forward -n recommender svc/recommender 8000:8000  # for the recommender

Be sure to add the necessary variables to your env. You can check that in the updated example env, but for convenience:

export CATALOGUE_URL="http://localhost:3030"
export RECOMMENDER_URL="http://localhost:8000"

Then as usual:

npm run dev

To-do

add participant fetching when selecting an offering
rework "back to search" button in selected offering page, to actually get back to the same search query (may become a separated issue)
implement streaming to improving catalogue search page loading experience (will very likely become a separate issue): covered by issue Implement streaming to load catalogue queries #39

Closes #1

…a from catalogue

…, featuring keywords list

xamcost

@AlvaroFernandezBejarano This PR is now ready for review! Sorry for the time it took, I got dragged away by other tasks. It implements all of what is mentioned in the original posts, yet to keep the development of catalogue browsing features manageable, I created a few issues for follow up work:

Implement streaming to load catalogue queries #39 to implement progressive loading of the catalogue upon querying it. It is not a priority, won't be addressed now.
Implement catalogue query filtering by providers and keywords #40 to implement the filtering by providers and keywords. This is the next step in the catalogue work.
Rework catalogue query pagination #41 to implement pagination. Only after Implement catalogue query filtering by providers and keywords #40
Make recommender optional #42 to make the use of the recommender optional: this is ultra low priority.

When you'll have time, please have a look and let me know!

xamcost · 2025-02-12T20:07:51Z

.github/workflows/build_push_docker_image.yaml

+          echo "CATALOGUE_URL=${{ secrets.CATALOGUE_URL }}" > .env.production
+          echo "RECOMMENDER_URL=${{ secrets.RECOMMENDER_URL }}" >> .env.production


Having these two env variables set is required for the production build to work, otherwise the prefetching of nextjs fails.

xamcost · 2025-02-12T20:09:05Z

Dockerfile

@@ -10,7 +12,7 @@ FROM node:20-alpine as builder
 WORKDIR /app
 COPY --from=deps /app/node_modules ./node_modules
 COPY . .
-RUN npm run build
+RUN npm run ${BUILD_CMD}


Remark: this is really an optional addition, to enable us to build locally or to do a development build with prefetching disabled. I am happy to remove it if you find that irrelevant.

xamcost · 2025-02-12T20:10:07Z

next.config.js

-        hostname: 'picsum.photos',
-        port: '',
-        pathname: '/200'
+        hostname: '**'


Not ideal in terms of security, but we have no choice if we want to be able to render any images stored in the offerings and participants.. if you have any ideas to make this more secure, let me know!

xamcost · 2025-02-12T20:11:18Z

package.json

+    "compile": "next build --experimental-build-mode compile",
+    "generate": "next build --experimental-build-mode generate",


This two commands are here to enable us to build the app without prefetching, therefore not requiring valid URLs for the catalogue or recommender. I am happy to remove that if you find this irrelevant.

xamcost · 2025-02-12T20:13:29Z

src/app/catalogue/[offeringId]/components/BackToSearchButton.jsx

+  const goBack = () => {
+    if (document.referrer.includes('/catalogue?')) {
+      window.history.back()
+    } else {
+      window.location.href = '/catalogue'
+    }
+  }


It may sound unusual to use window.history directly, but I found that more elegant than creating a hook to modify Nextjs' router, which doesn't store the navigation history.

xamcost · 2025-02-12T20:14:06Z

src/app/catalogue/[offeringId]/components/DatasetInfoComponent.jsx

        </div>
        <div className='pt-2'>
-          <PriceCard price={asset.price} />
+          <PriceCard price={offering?.price ?? 0} />


NB: price is not part of the Sedimark ontology for the time being.

xamcost · 2025-02-12T20:19:14Z

src/app/catalogue/[offeringId]/components/Recommender.jsx

@@ -8,11 +8,11 @@ function Recommender ({ recommendations }) {
      <div className='flex gap-4 overflow-x-auto p-4 w-full'>
        {recommendations.map((recommendation, index) => {
          return (
-            <Link key={`${recommendation.title}-${recommendation.created_at}-${index + 1}`} href='#'>
+            <Link key={`${recommendation.title.value}-${recommendation.created.value}-${index + 1}`} href={`/catalogue/${btoa(recommendation.offering.value)}`}>


NB: offering IDs are URLs... so to put them in the URL path, they are encoded in base64. I am open to other approaches!

xamcost · 2025-02-12T20:21:23Z

src/app/catalogue/page.js

  const query = searchParams?.query || ''
  const currentPage = Number(searchParams?.page) || 1

  return (
    <div className='bg-sedimark-dark-deep-blue'>
      <SearchBar />
      <Suspense
-        key={type + query + currentPage} // Ensures the Catalogue component is re-rendered when the search parameters change
+        key={query + currentPage} // Ensures the Catalogue component is re-rendered when the search parameters change


The type has been removed because we won't distinguish two types of offerings anymore in the project (just datasets, no services).

xamcost · 2025-02-12T20:26:48Z

src/components/footer/Footer.jsx

+      <div>
+        <Image src='/img/ukri-icon.png' height={48} width={149} alt='UKRI' />
+      </div>


Sorry this addition is not relevant in the PR, but that's something we need to do.

xamcost added 11 commits February 3, 2025 16:23

maint: remove query type and unused searchbar component

8a8836b

feat: put sparql catalogue (but no filters)

9c744b0

feat: fetch all offerings if no query provided

9c232c4

dev: update catalogue sparql query

a091cee

fix: offering item to match results returned by query

6be4471

dev: add functions to fetch providers and keywords from catalogue

3c5ad9a

feat: remove mockCatalogue, mockFilters and mockKeywords to fetch dat…

fb7f871

…a from catalogue

feat: first version of offering detail page with recommender

2e6ebdf

feat: add dedicated function to fetch offering details from catalogue…

cdd9d5f

…, featuring keywords list

maint: return list of offerings from catalogue directly

eb22d6e

dev: add schema.org in sparql prefixes

0d210f0

xamcost added Func: Offering Discovery Issues related to catalogue browsing Type: enhancement New feature or request labels Feb 6, 2025

xamcost added 5 commits February 12, 2025 13:21

feat: add fetching of provider data in offering details

9c0cbdc

dev: adjust participant logo size

62b090a

feat: add ukri icon in footer

eaae483

feat: add navigation to recommended offerings

d80d11e

feat: implements back to search button

4e411b7

This was referenced Feb 12, 2025

Implement streaming to load catalogue queries #39

Open

Rework catalogue query pagination #41

Open

Make recommender optional #42

Open

xamcost added 2 commits February 12, 2025 16:26

fix: nextjs prefetching causing docker image building to fail

7c8f6ce

fix: recommender url in production env

fe84423

xamcost marked this pull request as ready for review February 12, 2025 15:45

xamcost commented Feb 12, 2025

View reviewed changes

xamcost requested a review from AlvaroFernandezBejarano February 13, 2025 08:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(#1): catalogue and recommender integration #37

feat(#1): catalogue and recommender integration #37

xamcost commented Feb 6, 2025 •

edited

Loading

xamcost left a comment

xamcost Feb 12, 2025

xamcost Feb 12, 2025

xamcost Feb 12, 2025

xamcost Feb 12, 2025

xamcost Feb 12, 2025

xamcost Feb 12, 2025

xamcost Feb 12, 2025

xamcost Feb 12, 2025

xamcost Feb 12, 2025

		echo "CATALOGUE_URL=${{ secrets.CATALOGUE_URL }}" > .env.production
		echo "RECOMMENDER_URL=${{ secrets.RECOMMENDER_URL }}" >> .env.production

		"compile": "next build --experimental-build-mode compile",
		"generate": "next build --experimental-build-mode generate",

feat(#1): catalogue and recommender integration #37

Are you sure you want to change the base?

feat(#1): catalogue and recommender integration #37

Conversation

xamcost commented Feb 6, 2025 • edited Loading

Description

How to test

To-do

xamcost left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xamcost commented Feb 6, 2025 •

edited

Loading