feat: adds sortable fields to studio content search index [FC-0059] #35103

pomegranited · 2024-07-09T10:28:40Z

Description

This PR adds new datetime fields to the serialized library block metadata: created, modified, and last_published, and adds support for sorting search results by these dates or the block title/display_name.

Also adds last_published to the list of fields we can filter by, so that e.g. sorting by "Recently Published" can also filter out unpublished content.

This change should not impact any current users, as content libraries v2 are not yet ready for production use.

Supporting information

Part of openedx/frontend-app-authoring#1038

Testing instructions

See openedx/frontend-app-authoring#1147 for full testing instructions.

Deadline

ASAP -- to meet project deadlines

Other information

These changes can be deployed by running the management command:

python manage.py cms reindex_studio --experimental

* new datetime fields added to library component serializer: created, modified, and last_published * new datetime fields are stored in Meilisearch as POSIX timestamps for sorting * Meilisearch search ranking updated to allow sorting by display_name or these new datetime fields * Updates tests to verify these new properties and client settings

openedx-webhooks · 2024-07-09T10:28:45Z

Thanks for the pull request, @pomegranited!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently maintained by @openedx/wg-maintenance-edx-platform. Tag them in a comment and let them know that your changes are ready for review.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

* Instead of using `latest.modified` (which always queries the database), use `draft.latest` * However, we have to increase the query count by 1 query per block, because the published_at date can only be found in the publish log, which must be fetched individually for each block.

pomegranited · 2024-07-09T12:14:30Z

openedx/core/djangoapps/content_tagging/tests/test_objecttag_export_helpers.py

@@ -441,7 +441,7 @@ def test_build_library_object_tree(self) -> None:
        """
        Test if we can export a library
        """
-        with self.assertNumQueries(8):
+        with self.assertNumQueries(11):


My initial changes to LibraryXBlockMetadata.from_component brought this query count up to 14, because it used component.versioning.latest.

Using component.versioning.draft instead brought the query count down to 11, because the authoring API doesn't give us a more efficient way to fetch the published_at date than from the last_publish_log.

Welcome any advice on how to avoid the increased query count here!

This seems like an @ormsbee question if I'm correctly understanding that it's about efficiently fetching modified dates from learning core's authoring API.

yusuf-musleh

@pomegranited Great work on this! I've tested it out with the frontend and it's working quite well. I just left a comment the issue with updating the index when updating the component in the library. I shared an idea, hopefully it might help solve it.

openedx/core/djangoapps/content/search/handlers.py

rpenido

LGTM 👍
Thank you for your work @pomegranited!

I tested this using the instructions from feat: adds sort widget to search manager and library component page [FC-0059] frontend-app-authoring#1147
I read through the code
~~I checked for accessibility issues~~
Includes documentation

Note: While testing this, I found an issue (not related) updating my index. I create a PR to circumvent this:

fix: skip block indexing if it raises an error while loading #35139

…t-dates

openedx/core/djangoapps/xblock/rest_api/views.py

openedx-webhooks · 2024-07-25T18:18:51Z

@pomegranited 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

edx-pipeline-bot · 2024-07-26T07:26:17Z

2U Release Notice: This PR has been deployed to the edX production environment.

edx-pipeline-bot · 2024-07-26T07:26:17Z

2U Release Notice: This PR has been deployed to the edX production environment.

pomegranited · 2024-07-26T15:42:26Z

openedx/core/djangoapps/content_libraries/api.py

        return cls(
            usage_key=LibraryUsageLocatorV2(
                library_key,
                component.component_type.name,
                component.local_key,
            ),
            display_name=component.versioning.draft.title,
+            created=component.created,
+            modified=component.versioning.draft.created,
+            last_published=None if last_publish_log is None else last_publish_log.published_at,


Hi @ormsbee 👋
I added this code to record the published date for library components in the Studio search index. However, it doesn't work, seemingly because components don't get publish records recorded when they are published -- only the parent library does.

However, there isn't a date available on the component.versioning.published ComponentVersion -- the published.created matches the draft.modified date after publishing, and doesn't represent the actual date published.

What would be the best way to address this issue?

CC @bradenmacdonald

And openedx/openedx-learning#204

I don't remember if there's a good helper for it at the moment, but I think you can get the datetime when a Component was published by following these relations:

Component.publishable_entity -> PublishableEntity.published -> Published.publish_log_record -> PublishLogRecord.publish_log -> PublishLog. published_at

bradenmacdonald · 2024-07-30T17:00:08Z

@pomegranited I think one of the new tests might be flaky: https://github.com/openedx/frontend-app-course-authoring/actions/runs/10166343467/job/28116281335

Might need to wrap the failing line in a waitFor ?

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jul 9, 2024

pomegranited commented Jul 9, 2024

View reviewed changes

pomegranited requested a review from yusuf-musleh July 10, 2024 09:08

pomegranited mentioned this pull request Jul 10, 2024

feat: adds sort widget to search manager and library component page [FC-0059] openedx/frontend-app-authoring#1147

Merged

yusuf-musleh reviewed Jul 10, 2024

View reviewed changes

openedx/core/djangoapps/content/search/handlers.py Show resolved Hide resolved

Merge branch 'master' into jill/index-component-dates

b8bf0bb

yusuf-musleh mentioned this pull request Jul 12, 2024

feat: Add "Recently Modified" library section [FC-0059] openedx/frontend-app-authoring#1151

Closed

feat: raise LIBRARY_BLOCK_UPDATED in the xblock edit REST API

b7f8f8e

mphilbrick211 added the FC Relates to an Axim Funded Contribution project label Jul 16, 2024

rpenido approved these changes Jul 18, 2024

View reviewed changes

pomegranited added 4 commits July 20, 2024 08:26

Merge branch 'master' into jill/index-component-dates

ecf0765

fix: allow content search to filter on last_modified

74a7ac8

Merge remote-tracking branch 'origin/master' into jill/index-componen…

224ec03

…t-dates

Merge remote-tracking branch 'origin/master' into jill/index-componen…

0ffd91e

…t-dates

pomegranited requested a review from bradenmacdonald July 24, 2024 17:44

bradenmacdonald reviewed Jul 24, 2024

View reviewed changes

openedx/core/djangoapps/xblock/rest_api/views.py Outdated Show resolved Hide resolved

pomegranited added 2 commits July 26, 2024 02:21

refactor: use LearningContext to send "block updated" event

cc67395

Merge branch 'master' into jill/index-component-dates

c1f7b05

bradenmacdonald reviewed Jul 25, 2024

View reviewed changes

openedx/core/djangoapps/xblock/rest_api/views.py Outdated Show resolved Hide resolved

fix: nit, don't redeclare usage_key

ceaf314

bradenmacdonald approved these changes Jul 25, 2024

View reviewed changes

bradenmacdonald enabled auto-merge (squash) July 25, 2024 18:15

bradenmacdonald merged commit 6b5d812 into openedx:master Jul 25, 2024
48 checks passed

bradenmacdonald deleted the jill/index-component-dates branch July 25, 2024 18:18

pomegranited commented Jul 26, 2024

View reviewed changes

pomegranited mentioned this pull request Jul 27, 2024

fix: content search last_published date [FC-0059] #35195

Merged

bradenmacdonald mentioned this pull request Aug 1, 2024

fix: error when saving changes to a v2 library block [FC-0059] #35217

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: adds sortable fields to studio content search index [FC-0059] #35103

feat: adds sortable fields to studio content search index [FC-0059] #35103

pomegranited commented Jul 9, 2024 •

edited

Loading

openedx-webhooks commented Jul 9, 2024 •

edited

Loading

pomegranited Jul 9, 2024

bradenmacdonald Jul 24, 2024

yusuf-musleh left a comment

rpenido left a comment •

edited

Loading

openedx-webhooks commented Jul 25, 2024

edx-pipeline-bot commented Jul 26, 2024

edx-pipeline-bot commented Jul 26, 2024

pomegranited Jul 26, 2024

pomegranited Jul 27, 2024

pomegranited Jul 29, 2024

ormsbee Jul 30, 2024

bradenmacdonald commented Jul 30, 2024

feat: adds sortable fields to studio content search index [FC-0059] #35103

feat: adds sortable fields to studio content search index [FC-0059] #35103

Conversation

pomegranited commented Jul 9, 2024 • edited Loading

Description

Supporting information

Testing instructions

Deadline

Other information

openedx-webhooks commented Jul 9, 2024 • edited Loading

What's next?

🔘 Get product approval

🔘 Provide context

🔘 Get a green build

🔘 Let us know that your PR is ready for review:

Who will review my changes?

Where can I find more information?

When can I expect my changes to be merged?

pomegranited Jul 9, 2024

Choose a reason for hiding this comment

bradenmacdonald Jul 24, 2024

Choose a reason for hiding this comment

yusuf-musleh left a comment

Choose a reason for hiding this comment

rpenido left a comment • edited Loading

Choose a reason for hiding this comment

openedx-webhooks commented Jul 25, 2024

edx-pipeline-bot commented Jul 26, 2024

edx-pipeline-bot commented Jul 26, 2024

pomegranited Jul 26, 2024

Choose a reason for hiding this comment

pomegranited Jul 27, 2024

Choose a reason for hiding this comment

pomegranited Jul 29, 2024

Choose a reason for hiding this comment

ormsbee Jul 30, 2024

Choose a reason for hiding this comment

bradenmacdonald commented Jul 30, 2024

pomegranited commented Jul 9, 2024 •

edited

Loading

openedx-webhooks commented Jul 9, 2024 •

edited

Loading

rpenido left a comment •

edited

Loading