Skip to content
This repository has been archived by the owner on Jun 14, 2024. It is now read-only.

Commit

Permalink
feat: add support to new options_order settings attribute for multi…
Browse files Browse the repository at this point in the history
… label selection questions (#133)

# Description

This PR add changes so a new attribute with name `options_order` can be
set for multi selection question settings.

This attribute is an enumerable that can have the following values:
* `natural`: the default value, showing that the options/labels should
be ordered in their natural order where they have been specified for the
questions.
* `suggestion`: showing that options/labels should be ordered giving
priority to the labels that have suggestions (having or not an assigned
`score`) and sorting by `score` when available.

Some more details about this implementation:
- When retrieving a multi label selection question without
`options_order` in his settings, `options_order: natural` will be
returned inside the settings.
- When creating a multi label selection question if `options_order` has
not been defined as part of the creation body it will be inserted as
`options_order: natural` inside the database. Notice how a `None` value
is not accepted.
- When updating a multi label selection question `options_order` is
optional but `None` is not a valid value.

Closes #132 

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (change restructuring the codebase without changing
functionality)
- [ ] Improvement (change adding some improvement to an existing
functionality)
- [ ] Documentation update

**How Has This Been Tested**

(Please describe the tests that you ran to verify your changes. And
ideally, reference `tests`)

- [x] Adding new tests and improving old ones.

**Checklist**

- [ ] I added relevant documentation
- [ ] follows the style guidelines of this project
- [ ] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [ ] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)
  • Loading branch information
jfcalvo authored Apr 30, 2024
1 parent f83212a commit 1619a8e
Show file tree
Hide file tree
Showing 12 changed files with 320 additions and 6 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ These are the section headers that we use:
- Added support to specify a list of score values for suggestions `score` attribute. ([#98](https://github.com/argilla-io/argilla-server/pull/98))
- Added `GET /api/v1/settings` new endpoint exposing Argilla and Hugging Face settings when available. ([#127](https://github.com/argilla-io/argilla-server/pull/127))
- Added `ARGILLA_SHOW_HUGGINGFACE_SPACE_PERSISTANT_STORAGE_WARNING` new environment variable to disable warning message when Hugging Face Spaces persistent storage is disabled. ([#124](https://github.com/argilla-io/argilla-server/pull/124))
- Added `options_order` new settings attribute to support specify an order for options in multi label selection questions. ([#133](https://github.com/argilla-io/argilla-server/pull/133))

### Removed

Expand Down
1 change: 1 addition & 0 deletions src/argilla_server/contexts/questions.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ async def update_question(
QuestionUpdateValidator(question_update).validate_for(question)

params = question_update.dict(exclude_unset=True)

return await question.update(db, **params)


Expand Down
5 changes: 5 additions & 0 deletions src/argilla_server/enums.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,8 @@ class SortOrder(str, Enum):
class SimilarityOrder(str, Enum):
most_similar = "most_similar"
least_similar = "least_similar"


class OptionsOrder(str, Enum):
natural = "natural"
suggestion = "suggestion"
6 changes: 6 additions & 0 deletions src/argilla_server/schemas/v1/questions.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
from typing import Any, Dict, List, Literal, Optional, Union
from uuid import UUID

from argilla_server.enums import OptionsOrder
from argilla_server.models import QuestionType
from argilla_server.pydantic_v1 import BaseModel, Field, conlist, constr, root_validator, validator
from argilla_server.schemas.base import UpdateSchema
Expand Down Expand Up @@ -197,14 +198,19 @@ class LabelSelectionSettingsUpdate(UpdateSchema):
# Multi-label selection question
class MultiLabelSelectionQuestionSettings(LabelSelectionQuestionSettings):
type: Literal[QuestionType.multi_label_selection]
options_order: OptionsOrder = OptionsOrder.natural


class MultiLabelSelectionQuestionSettingsCreate(LabelSelectionQuestionSettingsCreate):
type: Literal[QuestionType.multi_label_selection]
options_order: OptionsOrder = OptionsOrder.natural


class MultiLabelSelectionQuestionSettingsUpdate(LabelSelectionSettingsUpdate):
type: Literal[QuestionType.multi_label_selection]
options_order: Optional[OptionsOrder]

__non_nullable_fields__ = {"options_order"}


# Ranking question
Expand Down
3 changes: 2 additions & 1 deletion tests/factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
import random

import factory
from argilla_server.enums import FieldType, MetadataPropertyType
from argilla_server.enums import FieldType, MetadataPropertyType, OptionsOrder
from argilla_server.models import (
Dataset,
Field,
Expand Down Expand Up @@ -353,6 +353,7 @@ class MultiLabelSelectionQuestionFactory(QuestionFactory):
{"value": "option2", "text": "Option 2", "description": None},
{"value": "option3", "text": "Option 3", "description": None},
],
"options_order": OptionsOrder.natural,
}


Expand Down
13 changes: 13 additions & 0 deletions tests/unit/api/v1/datasets/questions/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright 2021-present, the Recognai S.L. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
114 changes: 114 additions & 0 deletions tests/unit/api/v1/datasets/questions/test_create_dataset_question.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Copyright 2021-present, the Recognai S.L. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from uuid import UUID

import pytest
from argilla_server.enums import OptionsOrder, QuestionType
from argilla_server.models import Question
from httpx import AsyncClient
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession

from tests.factories import DatasetFactory


@pytest.mark.asyncio
class TestCreateDatasetQuestion:
def url(self, dataset_id: UUID) -> str:
return f"/api/v1/datasets/{dataset_id}/questions"

async def test_create_dataset_multi_label_selection_question_with_options_order(
self, db: AsyncSession, async_client: AsyncClient, owner_auth_header: dict
):
dataset = await DatasetFactory.create()

response = await async_client.post(
self.url(dataset.id),
headers=owner_auth_header,
json={
"name": "name",
"title": "title",
"settings": {
"type": QuestionType.multi_label_selection,
"options": [
{"value": "label-a", "text": "Label A"},
{"value": "label-b", "text": "Label B"},
],
"options_order": OptionsOrder.suggestion,
},
},
)

assert response.status_code == 201

response_json = response.json()
assert response_json["settings"]["options_order"] == OptionsOrder.suggestion

question = (await db.execute(select(Question).filter_by(id=UUID(response_json["id"])))).scalar_one()
assert question.settings["options_order"] == OptionsOrder.suggestion

async def test_create_dataset_multi_label_selection_question_without_options_order(
self, db: AsyncSession, async_client: AsyncClient, owner_auth_header: dict
):
dataset = await DatasetFactory.create()

response = await async_client.post(
self.url(dataset.id),
headers=owner_auth_header,
json={
"name": "name",
"title": "title",
"settings": {
"type": QuestionType.multi_label_selection,
"options": [
{"value": "label-a", "text": "Label A"},
{"value": "label-b", "text": "Label B"},
],
},
},
)

assert response.status_code == 201

response_json = response.json()
assert response_json["settings"]["options_order"] == OptionsOrder.natural

question = (await db.execute(select(Question).filter_by(id=UUID(response_json["id"])))).scalar_one()
assert question.settings["options_order"] == OptionsOrder.natural

async def test_create_dataset_multi_label_selection_question_with_options_order_as_none(
self, db: AsyncSession, async_client: AsyncClient, owner_auth_header: dict
):
dataset = await DatasetFactory.create()

response = await async_client.post(
self.url(dataset.id),
headers=owner_auth_header,
json={
"name": "name",
"title": "title",
"settings": {
"type": QuestionType.multi_label_selection,
"options": [
{"value": "label-a", "text": "Label A"},
{"value": "label-b", "text": "Label B"},
],
"options_order": None,
},
},
)

assert response.status_code == 422
assert (await db.execute(select(func.count(Question.id)))).scalar_one() == 0
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Copyright 2021-present, the Recognai S.L. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from uuid import UUID

import pytest
from argilla_server.enums import OptionsOrder, QuestionType
from httpx import AsyncClient

from tests.factories import QuestionFactory


@pytest.mark.asyncio
class TestListDatasetQuestions:
def url(self, dataset_id: UUID) -> str:
return f"/api/v1/datasets/{dataset_id}/questions"

async def test_list_dataset_multi_label_selection_question_with_options_order(
self, async_client: AsyncClient, owner_auth_header: dict
):
question = await QuestionFactory.create(
settings={
"type": QuestionType.multi_label_selection,
"options": [
{"value": "label-a", "text": "Label A"},
{"value": "label-b", "text": "Label B"},
],
"options_order": OptionsOrder.suggestion,
},
)

response = await async_client.get(self.url(question.dataset_id), headers=owner_auth_header)

assert response.status_code == 200
assert response.json()["items"][0]["settings"]["options_order"] == OptionsOrder.suggestion
assert question.settings["options_order"] == OptionsOrder.suggestion

async def test_list_dataset_multi_label_selection_question_without_options_order(
self, async_client: AsyncClient, owner_auth_header: dict
):
question = await QuestionFactory.create(
settings={
"type": QuestionType.multi_label_selection,
"options": [
{"value": "label-a", "text": "Label A"},
{"value": "label-b", "text": "Label B"},
],
},
)

response = await async_client.get(self.url(question.dataset_id), headers=owner_auth_header)

assert response.status_code == 200
assert response.json()["items"][0]["settings"]["options_order"] == OptionsOrder.natural
assert "options_order" not in question.settings
13 changes: 13 additions & 0 deletions tests/unit/api/v1/questions/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright 2021-present, the Recognai S.L. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
88 changes: 86 additions & 2 deletions tests/unit/api/v1/questions/test_update_question.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,13 @@
from uuid import UUID

import pytest
from argilla_server.enums import QuestionType
from argilla_server.enums import OptionsOrder, QuestionType
from argilla_server.models import Question
from httpx import AsyncClient
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession

from tests.factories import LabelSelectionQuestionFactory, SpanQuestionFactory, TextQuestionFactory
from tests.factories import LabelSelectionQuestionFactory, QuestionFactory, SpanQuestionFactory, TextQuestionFactory


@pytest.mark.asyncio
Expand Down Expand Up @@ -91,6 +94,87 @@ async def test_update_question_with_different_options(self, async_client: AsyncC
"detail": "the option values cannot be modified. found unexpected option values: ['label-a', 'label-b', 'label-c']"
}

async def test_update_multi_label_selection_question_with_options_order(
self, db: AsyncSession, async_client: AsyncClient, owner_auth_header: dict
):
question = await QuestionFactory.create(
settings={
"type": QuestionType.multi_label_selection,
"options": [
{"value": "label-a", "text": "Label A"},
{"value": "label-b", "text": "Label B"},
],
"options_order": OptionsOrder.natural,
}
)

response = await async_client.patch(
self.url(question.id),
headers=owner_auth_header,
json={
"settings": {
"type": QuestionType.multi_label_selection,
"options_order": OptionsOrder.suggestion,
},
},
)

assert response.status_code == 200
assert response.json()["settings"]["options_order"] == OptionsOrder.suggestion
assert question.settings["options_order"] == OptionsOrder.suggestion

async def test_update_multi_label_selection_question_without_options_order(
self, async_client: AsyncClient, owner_auth_header: dict
):
question = await QuestionFactory.create(
settings={
"type": QuestionType.multi_label_selection,
"options": [
{"value": "label-a", "text": "Label A"},
{"value": "label-b", "text": "Label B"},
],
"options_order": OptionsOrder.suggestion,
}
)

response = await async_client.patch(
self.url(question.id),
headers=owner_auth_header,
json={"type": QuestionType.multi_label_selection},
)

assert response.status_code == 200
assert response.json()["settings"]["options_order"] == OptionsOrder.suggestion
assert question.settings["options_order"] == OptionsOrder.suggestion

async def test_update_multi_label_selection_question_with_options_order_as_none(
self, async_client: AsyncClient, owner_auth_header: dict
):
question = await QuestionFactory.create(
settings={
"type": QuestionType.multi_label_selection,
"options": [
{"value": "label-a", "text": "Label A"},
{"value": "label-b", "text": "Label B"},
],
"options_order": OptionsOrder.natural,
}
)

response = await async_client.patch(
self.url(question.id),
headers=owner_auth_header,
json={
"settings": {
"type": QuestionType.multi_label_selection,
"options_order": None,
},
},
)

assert response.status_code == 422
assert question.settings["options_order"] == OptionsOrder.natural

async def test_update_question_with_more_visible_options_than_allowed(
self, async_client: AsyncClient, owner_auth_header: dict
):
Expand Down
2 changes: 2 additions & 0 deletions tests/unit/api/v1/test_datasets.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
from argilla_server.constants import API_KEY_HEADER_NAME
from argilla_server.enums import (
DatasetStatus,
OptionsOrder,
RecordInclude,
ResponseStatusFilter,
SimilarityOrder,
Expand Down Expand Up @@ -368,6 +369,7 @@ async def test_list_dataset_questions(self, async_client: "AsyncClient", owner_a
{"value": "b", "text": "b", "description": "b"},
],
"visible_options": None,
"options_order": OptionsOrder.natural,
},
),
],
Expand Down
Loading

0 comments on commit 1619a8e

Please sign in to comment.