feat: meilisearch backend #150

qasimgulzar · 2024-06-11T12:27:03Z

Under this PR I have implemented a search backend for meilisearch.

Below is a screenshot of "discover now" page it is working with meilisearch backend.

Thread : https://discuss.openedx.org/t/is-meilisearch-a-viable-upgrade-alternative-to-opensearch/12400/12

openedx-webhooks · 2024-06-11T12:27:08Z

Thanks for the pull request, @qasimgulzar!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently maintained by @openedx/openedx-unmaintained. Tag them in a comment and let them know that your changes are ready for review.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

bradenmacdonald

Does this PR make courseware search work using Meilisearch or just add a new "auto-suggest" search that works using Meilisearch? Or both?

bradenmacdonald · 2024-07-12T14:45:30Z

edxsearch/settings.py

@@ -26,7 +26,7 @@
 # This is just a container for running tests
 DEBUG = True

-ALLOWED_HOSTS = []
+ALLOWED_HOSTS = ['*']


I'm not sure when/if this settings file is used, but I don't think we should just set ALLOWED_HOSTS = ['*']

It is mostly used during development

bradenmacdonald · 2024-07-12T14:46:58Z

edxsearch/settings.py

+################### Using Meilisearch (Beta) ###################
+
+# Enable Studio search features (powered by Meilisearch) (beta, off by default)
+MEILISEARCH_ENABLED = False


I don't think it makes sense to have a setting called MEILISEARCH_ENABLED. Instead, to enable Meilisearch, one would just set SEARCH_ENGINE to the Meilisearch backend.

bradenmacdonald · 2024-07-12T14:47:41Z

search/api.py

+
+def _meilisearch_auto_suggest_search_api(term, course_id, limit=30):
+    """
+    Perform an auto-suggest search using the Elasticsearch search engine.


Suggested change

Perform an auto-suggest search using the Elasticsearch search engine.

Perform an auto-suggest search using the Meilisearch search engine.

bradenmacdonald · 2024-07-12T14:47:58Z

search/api.py

+
+def _meilisearch_auto_suggest_search_api(term, course_id, limit=30):
+    """
+    Perform an auto-suggest search using the Elasticsearch search engine.


What is an "auto-suggest search"? How is it different than a regular search?

It was a POC for implementation I am going to remove it

bradenmacdonald · 2024-07-12T14:48:59Z

search/api.py

+    client = meilisearch.Client(settings.MEILISEARCH_URL, settings.MEILISEARCH_API_KEY)
+
+    # Define the index name
+    index_name = settings.MEILISEARCH_INDEX_PREFIX + "studio_content"


This is searching the studio index. It doesn't make sense. It should get the index name from the subclass, and it definitely shouldn't be searching the studio content. For courseware search, it should be COURSEWARE_CONTENT_INDEX_NAME

bradenmacdonald · 2024-07-12T14:59:04Z

search/meilisearch.py

+        source["id"] = usage_key.block_id
+    except (Exception, InvalidKeyError) as ex:  # pylint: disable=broad-except
+        source["id"] = sanitize_id(source["id"])
+        log.info(f"{str(ex)} - {source['id']} - {type(ex)}")


This should be covered by tests, and we don't need logging this verbose.

bradenmacdonald · 2024-07-12T14:59:42Z

search/meilisearch.py

+
+def sanitized_id(source: dict, create_usage_key=True) -> dict:
+    """
+    Sanitize the Id key to avoid restricted objects


I don't think this function makes sense. Can you clarify what it's doing?

Meilisearch doesn't allow to put course-key in ID attribute so to make it compatible I implemented this function.

bradenmacdonald · 2024-07-12T15:00:43Z

search/meilisearch.py

+    return source
+
+
+def build_filter(key, val):


All of these functions will need extensive tests cases.

bradenmacdonald · 2024-07-12T15:01:21Z

search/meilisearch.py

+    @property
+    def mappings(self):
+        """
+        Get mapping of current index.


What are the "mappings" ?

bradenmacdonald · 2024-07-12T15:02:04Z

search/meilisearch.py

+    def __init__(self, index=None, options=None):
+        super().__init__(index)
+        MEILISEARCH_URL = getattr(settings, "MEILISEARCH_URL", 'http://127.0.0.1:7700')
+        MEILISEARCH_API_KEY = getattr(settings, "MEILISEARCH_API_KEY", "masterKey")


I don't know if we should be putting the default values here - we might get inconsistent defaults if we just define the defaults separately in each place where we read the setting.

e0d · 2024-07-24T15:51:27Z

Your branch is behind the base. I've pulled in changes from master as a merge commit which will update your branch and cause the tests to be re-run.

@qasimgulzar should you be listed under the Arbisoft entity CLA? Currently we have to run tests on your PR manually.

qasimgulzar · 2024-07-25T09:21:51Z

Yes please if you can add me under arbisoft CLA that will be great.

regisb · 2024-07-31T10:18:32Z

Yes, I confirm that @qasimgulzar is part of Arbisoft and should be added to the company CLA. Is there any step that we should pursue on our side?

mphilbrick211 · 2024-08-22T20:23:09Z

Hi @qasimgulzar! I see your CLA is all set. Do you plan to pursue this pull request?

qasimgulzar · 2024-08-22T20:25:10Z

I am working on another extension but we might need this for further testing.

qasimgulzar · 2024-10-04T10:34:33Z

Hi @e0d, I hope you are doing great. I will no longer be part of edly from 7, OCT 2024. could you please keep my CLA back to individual and remove me from arbisoft's CLA.

mphilbrick211 · 2024-10-24T20:45:22Z

Hi @qasimgulzar! Can this pull request be closed?

qasimgulzar · 2024-10-25T07:48:19Z

closing this PR, as it is not intended to merged.

feat: added new auto suggest endpoint.

ef60540

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jun 11, 2024

qasimgulzar added 2 commits June 11, 2024 18:14

fix: code improvements

c60f577

fix: code improvements

d2b92fc

qasimgulzar marked this pull request as draft June 11, 2024 13:28

fix: code improvements

f9d2de1

mphilbrick211 added the needs test run Author's first PR to this repository, awaiting test authorization from Axim label Jun 13, 2024

qasimgulzar added 2 commits June 13, 2024 15:19

🧪added testcase for new auto-suggest view.

62ac4b0

📍dependency added

b0e8109

qasimgulzar marked this pull request as ready for review June 13, 2024 10:38

qasimgulzar added 4 commits June 20, 2024 17:32

add meilisearch backend

1e4261a

fix: add meilisearch backend

4bccabb

fix: added searchable

edee7ae

fix: added searchable

e64a5bc

bradenmacdonald mentioned this pull request Jul 12, 2024

feat: meilisearch backend openedx/edx-platform#35026

Closed

bradenmacdonald reviewed Jul 12, 2024

View reviewed changes

qasimgulzar added 3 commits July 15, 2024 11:34

fix: remove auto suggest

3a240df

fix: remove auto suggest

2fe2800

fix: move index settings to LMS settings

4450a37

qasimgulzar changed the title ~~feat: added new auto suggest endpoint.~~ feat: meilisearch backend Jul 15, 2024

Merge branch 'master' into qasim/autosuggest-api

2073181

mphilbrick211 removed the needs test run Author's first PR to this repository, awaiting test authorization from Axim label Jul 24, 2024

qasimgulzar closed this Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: meilisearch backend #150

feat: meilisearch backend #150

qasimgulzar commented Jun 11, 2024 •

edited

Loading

openedx-webhooks commented Jun 11, 2024 •

edited

Loading

bradenmacdonald left a comment

bradenmacdonald Jul 12, 2024

qasimgulzar Jul 15, 2024

bradenmacdonald Jul 12, 2024

bradenmacdonald Jul 12, 2024

bradenmacdonald Jul 12, 2024

qasimgulzar Jul 15, 2024

bradenmacdonald Jul 12, 2024

bradenmacdonald Jul 12, 2024

bradenmacdonald Jul 12, 2024

qasimgulzar Jul 15, 2024

bradenmacdonald Jul 12, 2024

bradenmacdonald Jul 12, 2024

bradenmacdonald Jul 12, 2024

e0d commented Jul 24, 2024

qasimgulzar commented Jul 25, 2024

regisb commented Jul 31, 2024

mphilbrick211 commented Aug 22, 2024

qasimgulzar commented Aug 22, 2024 •

edited

Loading

qasimgulzar commented Oct 4, 2024

mphilbrick211 commented Oct 24, 2024

qasimgulzar commented Oct 25, 2024

	Perform an auto-suggest search using the Elasticsearch search engine.
	Perform an auto-suggest search using the Meilisearch search engine.

feat: meilisearch backend #150

feat: meilisearch backend #150

Conversation

qasimgulzar commented Jun 11, 2024 • edited Loading

openedx-webhooks commented Jun 11, 2024 • edited Loading

What's next?

🔘 Get product approval

🔘 Provide context

🔘 Get a green build

🔘 Let us know that your PR is ready for review:

Who will review my changes?

Where can I find more information?

When can I expect my changes to be merged?

bradenmacdonald left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e0d commented Jul 24, 2024

qasimgulzar commented Jul 25, 2024

regisb commented Jul 31, 2024

mphilbrick211 commented Aug 22, 2024

qasimgulzar commented Aug 22, 2024 • edited Loading

qasimgulzar commented Oct 4, 2024

mphilbrick211 commented Oct 24, 2024

qasimgulzar commented Oct 25, 2024

qasimgulzar commented Jun 11, 2024 •

edited

Loading

openedx-webhooks commented Jun 11, 2024 •

edited

Loading

qasimgulzar commented Aug 22, 2024 •

edited

Loading