Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add language auto-tagging #32907

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
47af3bf
chore: update requirements
rpenido Aug 23, 2023
4ed15f5
feat: Add retrieve object_tags REST API (#577)
yusuf-musleh Aug 22, 2023
2bc5070
feat: add content auto-tagging (language)
rpenido Aug 23, 2023
0f62355
chore: add __init__.py
rpenido Aug 23, 2023
04b6282
Merge branch 'master' into rpenido/fal-3460-content-auto-tagging-by-s…
rpenido Aug 23, 2023
b06e2bb
style: fix pep8
rpenido Aug 23, 2023
d6bc300
style: fix pylint
rpenido Aug 23, 2023
9017818
style: fix pep8
rpenido Aug 23, 2023
ee179b1
test: fix query count and comments
rpenido Aug 25, 2023
858f263
test: fix query count
rpenido Aug 25, 2023
ea2e149
fix: add try..except
rpenido Aug 25, 2023
fe05c57
fix: fix exit condition to avoid error
rpenido Aug 25, 2023
5bddde6
test: fix query count
rpenido Aug 25, 2023
e0d816a
Merge branch 'master' into rpenido/fal-3460-content-auto-tagging-by-s…
rpenido Aug 25, 2023
56300b7
refactor: change fixture to migration
rpenido Aug 25, 2023
901f4d3
test: fix setup
rpenido Aug 25, 2023
09dea58
chore: trigger CD/CI
rpenido Aug 26, 2023
5e7b097
test: trying to fix race condition
rpenido Aug 26, 2023
a08f08a
test: fix query count
rpenido Aug 26, 2023
08cd8b5
style: fixed pylint
rpenido Aug 26, 2023
778552a
docs: fix docstring from review
rpenido Aug 28, 2023
975edfd
test: remove override_settings
rpenido Aug 28, 2023
70650e9
refactor: add typings
rpenido Aug 28, 2023
75f4b70
style: add typings
rpenido Aug 28, 2023
e6dbd52
fix: import annotations
rpenido Aug 28, 2023
7097dd4
test: refactor tests
rpenido Aug 28, 2023
2a6bcaf
refactor: fix tests and cleaning code
rpenido Aug 29, 2023
fabb4aa
revert: revert changes in api
rpenido Aug 29, 2023
7adcd8b
refactor: cleaning code
rpenido Aug 29, 2023
9f5f538
fix: pylint
rpenido Aug 29, 2023
c9e77e3
fix: patch tasks.modulestore and change replicaSet config
rpenido Aug 29, 2023
7213778
fix: pylint
rpenido Aug 29, 2023
b66a51b
test: change scope
rpenido Aug 29, 2023
28c6b88
fix: import
rpenido Aug 29, 2023
3021492
test: add replicaset config for lms
rpenido Aug 29, 2023
14db0ef
refactor: cleaning code
rpenido Aug 29, 2023
5f452b4
test: fix query count
rpenido Aug 29, 2023
5421f87
test: fix query count
rpenido Aug 29, 2023
f711c61
fix: remove Org Taxonomy in migration
rpenido Aug 30, 2023
6df7303
style: add empty space
rpenido Aug 30, 2023
ce23eb1
refactor: rename _update_tags to _set_initial_language_tag
rpenido Aug 30, 2023
eb9f2c0
refactor: fix types
rpenido Aug 30, 2023
a99b725
fix: remove ContentOrganizationTaxonomy
rpenido Aug 30, 2023
f174ece
feat: use default language if course.language not found
rpenido Aug 30, 2023
2ccdeb5
Revert "test: add replicaset config for lms"
rpenido Aug 30, 2023
1c27f65
feat: add CONTENT_TAGGING_AUTO WaffleSwitch
rpenido Aug 30, 2023
5200966
revert: revert changes in test_mixed_modulestore
rpenido Aug 30, 2023
2612fe9
style: remove unused import
rpenido Aug 30, 2023
5fc811d
test: add invalid tags tests
rpenido Aug 31, 2023
154a4ae
refactor: change WaffleSwitch to CourseWaffleFlag
rpenido Aug 31, 2023
6efc2c2
Merge branch 'master' into rpenido/fal-3460-content-auto-tagging-by-s…
rpenido Aug 31, 2023
1a1b634
test: fix query count
rpenido Aug 31, 2023
68c911a
Merge branch 'master' into rpenido/fal-3460-content-auto-tagging-by-s…
rpenido Sep 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions openedx/features/content_tagging/api.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
"""
Content Tagging APIs
"""
from typing import Iterator, List, Type, Union
from __future__ import annotations

from typing import Iterator, List, Type

import openedx_tagging.core.tagging.api as oel_tagging
from django.db.models import QuerySet
from opaque_keys.edx.keys import LearningContextKey
from opaque_keys.edx.locator import BlockUsageLocator
from opaque_keys.edx.keys import CourseKey, UsageKey
from openedx_tagging.core.tagging.models import Taxonomy
from organizations.models import Organization

Expand Down Expand Up @@ -117,9 +118,9 @@ def get_content_tags(

def tag_content_object(
taxonomy: Taxonomy,
tags: List,
object_id: Union[BlockUsageLocator, LearningContextKey],
) -> List[ContentObjectTag]:
tags: list,
object_id: CourseKey | UsageKey,
) -> list[ContentObjectTag]:
"""
This is the main API to use when you want to add/update/delete tags from a content object (e.g. an XBlock or
course).
Expand Down Expand Up @@ -150,4 +151,5 @@ def tag_content_object(
get_taxonomy = oel_tagging.get_taxonomy
get_taxonomies = oel_tagging.get_taxonomies
get_tags = oel_tagging.get_tags
delete_object_tags = oel_tagging.delete_object_tags
resync_object_tags = oel_tagging.resync_object_tags
4 changes: 4 additions & 0 deletions openedx/features/content_tagging/apps.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,7 @@ class ContentTaggingConfig(AppConfig):

default_auto_field = "django.db.models.BigAutoField"
name = "openedx.features.content_tagging"

def ready(self):
# Connect signal handlers
from . import handlers # pylint: disable=unused-import
22 changes: 0 additions & 22 deletions openedx/features/content_tagging/fixtures/system_defined.yaml

This file was deleted.

76 changes: 76 additions & 0 deletions openedx/features/content_tagging/handlers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
"""
Automatic tagging of content
"""

import logging

from django.dispatch import receiver
from openedx_events.content_authoring.data import CourseData, XBlockData
from openedx_events.content_authoring.signals import COURSE_CREATED, XBLOCK_CREATED, XBLOCK_DELETED, XBLOCK_UPDATED

from .tasks import delete_course_tags
from .tasks import (
delete_xblock_tags,
update_course_tags,
update_xblock_tags
)
from .toggles import CONTENT_TAGGING_AUTO

log = logging.getLogger(__name__)


@receiver(COURSE_CREATED)
def auto_tag_course(**kwargs):
"""
Automatically tag course based on their metadata
"""
course_data = kwargs.get("course", None)
if not course_data or not isinstance(course_data, CourseData):
log.error("Received null or incorrect data for event")
return

if not CONTENT_TAGGING_AUTO.is_enabled(course_data.course_key):
return

update_course_tags.delay(str(course_data.course_key))


@receiver(XBLOCK_CREATED)
@receiver(XBLOCK_UPDATED)
def auto_tag_xblock(**kwargs):
"""
Automatically tag XBlock based on their metadata
"""
xblock_info = kwargs.get("xblock_info", None)
if not xblock_info or not isinstance(xblock_info, XBlockData):
log.error("Received null or incorrect data for event")
return

if not CONTENT_TAGGING_AUTO.is_enabled(xblock_info.usage_key.course_key):
return

if xblock_info.block_type == "course":
# Course update is handled by XBlock of course type
update_course_tags.delay(str(xblock_info.usage_key.course_key))

update_xblock_tags.delay(str(xblock_info.usage_key))


@receiver(XBLOCK_DELETED)
def delete_tag_xblock(**kwargs):
"""
Automatically delete XBlock auto tags.
"""
xblock_info = kwargs.get("xblock_info", None)
if not xblock_info or not isinstance(xblock_info, XBlockData):
log.error("Received null or incorrect data for event")
return

if not CONTENT_TAGGING_AUTO.is_enabled(xblock_info.usage_key.course_key):
return

if xblock_info.block_type == "course":
# Course deletion is handled by XBlock of course type
delete_course_tags.delay(str(xblock_info.usage_key.course_key))

delete_xblock_tags.delay(str(xblock_info.usage_key))
Original file line number Diff line number Diff line change
@@ -1,37 +1,62 @@
# Generated by Django 3.2.20 on 2023-07-11 22:57

from django.db import migrations
from django.core.management import call_command
from openedx.features.content_tagging.models import ContentLanguageTaxonomy


def load_system_defined_taxonomies(apps, schema_editor):
"""
Creates system defined taxonomies
"""
"""

# Create system defined taxonomy instances
call_command('loaddata', '--app=content_tagging', 'system_defined.yaml')
Taxonomy = apps.get_model("oel_tagging", "Taxonomy")
author_taxonomy = Taxonomy(
pk=-2,
name="Content Authors",
description="Allows tags for any user ID created on the instance.",
enabled=True,
required=True,
allow_multiple=False,
allow_free_text=False,
visible_to_authors=False,
)
ContentAuthorTaxonomy = apps.get_model("content_tagging", "ContentAuthorTaxonomy")
author_taxonomy.taxonomy_class = ContentAuthorTaxonomy
author_taxonomy.save()

org_taxonomy = Taxonomy(
pk=-3,
name="Organizations",
description="Allows tags for any organization ID created on the instance.",
bradenmacdonald marked this conversation as resolved.
Show resolved Hide resolved
enabled=True,
required=True,
allow_multiple=False,
allow_free_text=False,
visible_to_authors=False,
)
ContentOrganizationTaxonomy = apps.get_model("content_tagging", "ContentOrganizationTaxonomy")
org_taxonomy.taxonomy_class = ContentOrganizationTaxonomy
org_taxonomy.save()

# Adding taxonomy class to the language taxonomy
Taxonomy = apps.get_model('oel_tagging', 'Taxonomy')
language_taxonomy = Taxonomy.objects.get(id=-1)
ContentLanguageTaxonomy = apps.get_model("content_tagging", "ContentLanguageTaxonomy")
language_taxonomy.taxonomy_class = ContentLanguageTaxonomy
language_taxonomy.save()


def revert_system_defined_taxonomies(apps, schema_editor):
"""
Deletes all system defined taxonomies
"""
Taxonomy = apps.get_model('oel_tagging', 'Taxonomy')
Taxonomy = apps.get_model("oel_tagging", "Taxonomy")
Taxonomy.objects.get(id=-2).delete()
Taxonomy.objects.get(id=-3).delete()


class Migration(migrations.Migration):

dependencies = [
('content_tagging', '0002_system_defined_taxonomies'),
("content_tagging", "0002_system_defined_taxonomies"),
]

operations = [
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
from django.db import migrations


def load_system_defined_org_taxonomies(apps, _schema_editor):
"""
Associates the system defined taxonomy Language (id=-1) to all orgs and
removes the ContentOrganizationTaxonomy (id=-3) from the database
"""
TaxonomyOrg = apps.get_model("content_tagging", "TaxonomyOrg")
TaxonomyOrg.objects.create(id=-1, taxonomy_id=-1, org=None)

Taxonomy = apps.get_model("oel_tagging", "Taxonomy")
Taxonomy.objects.get(id=-3).delete()




def revert_system_defined_org_taxonomies(apps, _schema_editor):
"""
Deletes association of system defined taxonomy Language (id=-1) to all orgs and
creates the ContentOrganizationTaxonomy (id=-3) in the database
"""
TaxonomyOrg = apps.get_model("content_tagging", "TaxonomyOrg")
TaxonomyOrg.objects.get(id=-1).delete()

Taxonomy = apps.get_model("oel_tagging", "Taxonomy")
org_taxonomy = Taxonomy(
pk=-3,
name="Organizations",
description="Allows tags for any organization ID created on the instance.",
enabled=True,
required=True,
allow_multiple=False,
allow_free_text=False,
visible_to_authors=False,
)
ContentOrganizationTaxonomy = apps.get_model("content_tagging", "ContentOrganizationTaxonomy")
org_taxonomy.taxonomy_class = ContentOrganizationTaxonomy
org_taxonomy.save()


class Migration(migrations.Migration):
dependencies = [
("content_tagging", "0003_system_defined_fixture"),
]

operations = [
migrations.RunPython(load_system_defined_org_taxonomies, revert_system_defined_org_taxonomies),
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Generated by Django 3.2.20 on 2023-08-30 15:17

from django.db import migrations


class Migration(migrations.Migration):

dependencies = [
('content_tagging', '0004_system_defined_org'),
]

operations = [
migrations.DeleteModel(
name='ContentOrganizationTaxonomy',
),
migrations.DeleteModel(
name='OrganizationModelObjectTag',
),
]
1 change: 0 additions & 1 deletion openedx/features/content_tagging/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,4 @@
from .system_defined import (
ContentLanguageTaxonomy,
ContentAuthorTaxonomy,
ContentOrganizationTaxonomy,
)
10 changes: 5 additions & 5 deletions openedx/features/content_tagging/models/base.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""
Content Tagging models
"""
from typing import List, Union
from __future__ import annotations

from django.db import models
from django.db.models import Exists, OuterRef, Q, QuerySet
Expand Down Expand Up @@ -49,7 +49,7 @@ class Meta:

@classmethod
def get_relationships(
cls, taxonomy: Taxonomy, rel_type: RelType, org_short_name: Union[str, None] = None
cls, taxonomy: Taxonomy, rel_type: RelType, org_short_name: str | None = None
) -> QuerySet:
"""
Returns the relationships of the given rel_type and taxonomy where:
Expand All @@ -68,7 +68,7 @@ def get_relationships(
@classmethod
def get_organizations(
cls, taxonomy: Taxonomy, rel_type: RelType
) -> List[Organization]:
) -> list[Organization]:
"""
Returns the list of Organizations which have the given relationship to the taxonomy.
"""
Expand All @@ -91,7 +91,7 @@ class Meta:
proxy = True

@property
def object_key(self) -> Union[BlockUsageLocator, LearningContextKey]:
def object_key(self) -> BlockUsageLocator | LearningContextKey:
"""
Returns the object ID parsed as a UsageKey or LearningContextKey.
Raises InvalidKeyError object_id cannot be parse into one of those key types.
Expand All @@ -115,7 +115,7 @@ class ContentTaxonomyMixin:
def taxonomies_for_org(
cls,
queryset: QuerySet,
org: Organization = None,
org: Organization | None = None,
) -> QuerySet:
"""
Filters the given QuerySet to those ContentTaxonomies which are available for the given organization.
Expand Down
48 changes: 0 additions & 48 deletions openedx/features/content_tagging/models/system_defined.py
Original file line number Diff line number Diff line change
@@ -1,62 +1,14 @@
"""
System defined models
"""
from typing import Type

from openedx_tagging.core.tagging.models import (
ModelSystemDefinedTaxonomy,
ModelObjectTag,
UserSystemDefinedTaxonomy,
LanguageTaxonomy,
)

from organizations.models import Organization
from .base import ContentTaxonomyMixin


class OrganizationModelObjectTag(ModelObjectTag):
"""
ObjectTags for the OrganizationSystemDefinedTaxonomy.
"""

class Meta:
proxy = True

@property
def tag_class_model(self) -> Type:
"""
Associate the organization model
"""
return Organization

@property
def tag_class_value(self) -> str:
"""
Returns the organization name to use it on Tag.value when creating Tags for this taxonomy.
"""
return "name"


class ContentOrganizationTaxonomy(ContentTaxonomyMixin, ModelSystemDefinedTaxonomy):
"""
Organization system-defined taxonomy that accepts ContentTags

Side note: The organization of an object is already encoded in its usage ID,
but a Taxonomy with Organization as Tags is being used so that the objects can be
indexed and can be filtered in the same tagging system, without any special casing.
"""

class Meta:
proxy = True

@property
def object_tag_class(self) -> Type:
"""
Returns OrganizationModelObjectTag as ObjectTag subclass associated with this taxonomy.
"""
return OrganizationModelObjectTag


class ContentLanguageTaxonomy(ContentTaxonomyMixin, LanguageTaxonomy):
"""
Language system-defined taxonomy that accepts ContentTags
Expand Down
Loading
Loading