Problems sending senses to api_update_gloss

[rem0g]

I am noticing a lot of errors regarding this gloss:

While submitting request to update phonology I am getting error Database locked at HUREN-C.

WHen submitting empty phonology feature such as:

"Handshape Change": ""

The handshape change is still not modified to empty value.

[susanodd]

It hasn't been changed. It's still the same.

(There are some other issues about the database getting locked.)

You want the empty value to erase the value?
There might be something to do with that, I will check.
You might need to use "-" as the value. Not an empty string.

[rem0g]

Ok, using "-" works! Thank you very much.

About gloss HUREN-C;

TransactionManagementError at /dictionary/api_update_gloss/5/49249/
An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

This is what i have sent:

{"Handedness": "N/A", "Strong Hand": "-", "Weak Hand": "B", "Handshape Change": "-", "Relation Between Articulators": "-", "Location": "-", "Contact Type": "-", "Movement Shape": "-", "Movement Direction": "-", "Repeated Movement": "", "Alternating Movement": "", "Relative Orientation: Movement": "-", "Relative Orientation: Location": "-", "Orientation Change": "-", "Virtual Object": "-", "Phonology Other": "-", "Mouth Gesture": "-", "Mouthing": "-", "Phonetic Variation": "-", "Senses": "{"en": [["hire"], ["rent, lease"]], "nl": [["huren"], ["huren"]]}"}

[susanodd]

Repeated Movement and Alternating Movement are supposed to be Booleans.

from the code:

if new_value not in ['true', 'True', 'TRUE', 'false', 'False', 'FALSE', 'Neutral', 'None']:

The atomic block error would be caused by other errors. It can't do a save because probably some other things are saved.
(It usually gives that kind of error if it can't do a last save because other objects have not been saved.)
Try modifying those fields to "False" to see if it goes away.

(I'll have to try to simulate your error with extra print statements to see where it is blocking.)
It could be causing that error for example if the error wasn't caught properly (bug) and then it didn't save the gloss, but then tries to update the timestamp. Probably a try-except is missing on a save. It's supposed to roll-back if an exception occurs. But that is also a problem because there are so many objects. That the Boolean field value is wrong should cause the method to report errors and not update at all.

[rem0g]

Phonology Other fields dont seem to be updated too.

Modifing the values to the ones you mentioned didnt help, it is causing only by this gloss but it could happen at other glosses too.

[susanodd]

Phonology Other fields dont seem to be updated too.

Modifing the values to the ones you mentioned didnt help, it is causing only by this gloss but it could happen at other glosses too.

Does your error message mention a specific line of code?

It's true that if it finds an error in the input fields, it will not do the update. (It's implemented as all or nothing, otherwise the transaction would be inconsistent.)

Do you want Handedness to be N/A ? There is the Weak Hand set, so is that Handedness 1 or is it something else?
(Some of the query code excludes Handedness N/A from queries. Like for minimal pairs.)
(I don't know if there's anything for multiple handedness fields. That's not supported yet. Is that needed?)

[susanodd]

These fields:

"Virtual Object": "-", "Phonology Other": "-", "Mouth Gesture": "-", "Mouthing": "-", "Phonetic Variation": "-",

are strings, so you can just put empty strings (the ones that are choices need to be "-")

If you don't want to set any value, you can exclude them from the json dictionary.
(It is ambiguous as to whether an empty value is a clear or to just ignore the field. So if you want to ignore the field and leave it as it is, don't include it in the update.)
I will check this again based on the logs.

[susanodd]

I looked up the gloss on Signbank.
It looks like the gloss doesn't have any Phonology yet.
Can you remove the fields from the update that stay "empty" from the update json fields?
So it only includes fields that need to be set.
Then make Handedness be 1 and set the Strong Hand not the Weak Hand.
As far as I know, if Handedness is 1, then the Strong Hand is set. (@uklomp can confirm this.)
You should also remove the Senses fields since you are not changing those.

[susanodd]

I can't find any errors related to the HUREN gloss. So the code thinks it's working correctly.
I can merely see that you tried the update url numerous times. But it is not raising any errors in the log.
Farther back, I find this one:

Internal Server Error: /dictionary/api_update_gloss/5/49249/
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'at[pid: 283|app: 0|req: 5442/11576] 10.208.155.251 () {62 vars in 1118 bytes} [Wed Jul 24 16:56:02 2024] POST /dictionary/api_update_gloss/5/49249/ => generated 282396 bytes in 517 msecs (HTTP/1.1 500) 5 headers in 178 bytes (1 switches on core 3)

(That's all it reports, nothing about what specifically. I'll add that exception to the code in order to catch it explicitly.)

In the logs, there is an error for ANTONI GAUDI

  File "/usr/lib/python3.12/urllib/request.py", line 1344, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.12/http/client.py", line 1331, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1342, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib/python3.12/http/client.py", line 1179, in putrequest
    self._output(self._encode_request(request))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 1260, in _encode_request
    return request.encode('ascii')
           ^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'ascii' codec can't encode character '\xcd' in position 15: ordinal not in range(128)

The only place in the code where it explicitly uses "ascii" is in the __str__ method of GlossVideoHistory.
The code saves the API video operations to the history. So that is what is causing that error. Probably that ought to be utf-8.

For the zip files, you don't need to put the gloss annotation in the zipped filename. It can be something else, like videos_archive.zip
It gets the annotation from inside the unzipped folder.

[rem0g]

There is definitely something wrong with HUREN-C, as i am trying to send this:

{"Handedness": "1", "Strong Hand": "-", "Weak Hand": "-", "Handshape Change": "-", "Relation Between Articulators": "-", "Location": "-", "Contact Type": "-", "Movement Shape": "-", "Movement Direction": "-", "Repeated Movement": "False", "Alternating Movement": "False", "Relative Orientation: Movement": "-", "Relative Orientation: Location": "-", "Orientation Change": "-", "Virtual Object": "-", "Phonology Other": "-", "Mouth Gesture": "-", "Mouthing": "-", "Phonetic Variation": "-", "Senses": "{"en": [["hire"], ["rent, lease"]], "nl": [["huren"], ["huren"]]}"}

Interestingly, the update goes through when i send like this:

{"Handedness": "1", "Strong Hand": "-", "Weak Hand": "-", "Handshape Change": "-", "Relation Between Articulators": "-", "Location": "-", "Contact Type": "-", "Movement Shape": "-", "Movement Direction": "-", "Repeated Movement": "False", "Alternating Movement": "False", "Relative Orientation: Movement": "-", "Relative Orientation: Location": "-", "Orientation Change": "-", "Virtual Object": "-", "Phonology Other": "-", "Mouth Gesture": "-", "Mouthing": "-", "Phonetic Variation": "-"}

So that's JSON payload without Senses.

Something is going wrong with Senses field?

[susanodd]

Okay, that helps us to know!

[susanodd]

@rem0g I found the problem. This has to do with the use of two different formats for representing the senses internally.

The json package considers them language-related fields and they are described as "Senses: Dutch"
but other code is using that "nl" dictionary format and storing them together as the right hand of Senses.

I wrote a new function to test whether the senses field in the update differs from that in the database.

You need to omit the quotes around the Senses right-hand side. It's already in json. (It's not a string.)

"Senses": {"en": [["hire"], ["rent, lease"]], "nl": [["huren"], ["huren"]]}

This has been deployed.

[rem0g]

It works again! Thank you

[Jetske]

@rem0g One more thing relevant to this issue:
We noticed that you send all the fields in the request to update the gloss. However this gives a lot of non-informative changes in the revision history (see https://signbank.cls.ru.nl/dictionary/gloss/49249/history)

We clearly did not document this well ;)
However the intention for this api was to only include the fields that are changed so it only updates those and leaves the others as is and they are not added to the revision history. Just so you know! You can close this issue again when read.

[rem0g]

Senses update doesnt work anymore again:

Payload:

{
  "Senses": {
    "en": [
      [
        "string"
      ]
    ],
    "nl": [
      [
        "string"
      ]
    ]
  }
}

AttributeError at /dictionary/api_update_gloss/5/47662/

'dict' object has no attribute 'strip'

[susanodd]

The format is wrong that you show above.

@rem0g can you please stop putting "empty" "not changed" things in the "update" request?

The gloss you show has hundreds of entries in the revision history.

https://signbank.cls.ru.nl/dictionary/gloss/47662/history

That causes extreme payload and causes the database to get locked. Hundreds of comparisons are being made to check if things are changed and hundreds of objects are being created for all the revision histories.

It is likely that something else is causing the problem or the operation is failing because the database is getting locked.

Think "thrashing" (too many operations are competing for resources)

Every API request is a transaction. So the more "empty" things you put that need to be processed the more computations need to be done inside that transaction.

[rem0g]

Can you please explain why is the format wrong? It's the format i got from the API documentation:

https://signbank.github.io/Global-signbank/#/Updating%20Signbank%20data/post_dictionary_api_update_gloss__datasetid___glossid__

With omitted strings as you mentioned before.

About the empty requests, yes I will be soon working on that soon. :)

[susanodd]

I looks like you have return characters in it?

The processing needs to parse it, so it should be without pretty printing. (It's a Python library that parses it.)

Normally, "strip" is done to make sure there are no spaces in the input.

Can you post your actual input value for the field? I will find out if there is a parsing error.

[susanodd]

The fix isn't live yet!!!

[susanodd]

This is incorrect:

{'Senses': {'en': '[["hire"], ["rent, lease"]]',
            'nl': '[["huren"], ["huren"]]'}}

There should not be quotes around this:

'[["hire"], ["rent, lease"]]'

The length of the lists for each language are compared. It's a list.

[susanodd]

I just deployed and restarted signbank now.

[susanodd]

Please ignore what is in the Swagger description. It is misleading.

There used to be examples in a separate page in the Wiki. That page was deleted by @Woseseltops. We can't get it back.

[rem0g]

Still not working, sorry.

Please send me your code of how you update your Senses including the API connection to api_update_gloss :)

[susanodd]

There's some example code in the html temptates (used for debugging)

signbank/dictionary/templates/dictionary/

signbank/dictionary/templates/dictionary/virtual_machine_gloss_update_api.html
signbank/dictionary/templates/dictionary/virtual_machine_gloss_update_api_token_en.html
signbank/dictionary/templates/dictionary/virtual_machine_gloss_update_api_token_nl.html

The ones that say "token" in the filename you can open locally in the browser using "file://"
(All the Django has been removed.)
(Those have the dataset id and the gloss id hard coded in the file. You want to change that.)

The one that doesn't say token (first one), you can open in Signbank using

dictionary/test_am_update_gloss/5/49249/

(using the dataset ID and the gloss ID in the url. There is a template set up in views.py that just allows that to be processed.
Similar templates exist for other API testing.)

The first one:
signbank/dictionary/templates/dictionary/virtual_machine_gloss_update_api.html

you can set the interface language and the field names change language.

There are some other templates for video files, But those have become obsolete when we started to use Swagger.

[susanodd]

@rem0g I should note that to actually UPDATE the senses, there is still a bug.

A new issue has been made for that #1461 (I'm working on that now, but I need help from @vanlummelhuizen for the signal that has a bug. When a sense is deleted, a signal is called that renumbers the remaining senses. But we don't want to do this, because the API is going to replace all the senses and create them again. The signal is where the bug is.)

The bug you found has to do with the "input" part of the request.

[rem0g]

Using https://github.com/Signbank/Global-signbank/blob/master/signbank/dictionary/templates/dictionary/virtual_machine_gloss_update_api_token_en.html is giving me same HTTP 500 error with this input in the field senses:

{"en": [["hire"], ["rent, lease"]], "nl": [["huren"], ["huren"]]}

[susanodd]

Does it give any other information in the browser?

You are using it as file://......

[susanodd]

There is a consistency constraint violation in this part:

"nl": [["huren"], ["huren"]]}

because they are identical.

I don't know whether that is correct or if that's a bug in the code. I thought it was allowed to use the same keywords if the matching translation if the other language is different.

[rem0g]

I have modified the console output as it produces HTTP 500 error in html code:

<pre class="exception_value">&#x27;dict&#x27; object has no attribute &#x27;strip&#x27;</pre>

When using

{"en": [["hire"], ["rent, lease"]], "nl": [["huren"], ["huren"]]}

URL:         url : url + "/dictionary/api_update_gloss/" + datasetid + '/' + glossid + '/',
    var url = 'https://signbank.cls.ru.nl/';
    var datasetid = '5';
    var glossid = '45834';

Can you please post your script when you have managed to update your Senses via Python using requests.post Python module? I am curious to how you have managed to update it.

[rem0g]

This is mine:

import requests
import json
from bs4 import BeautifulSoup
headers = {
    "Authorization": "Bearer xxxxx",
    'Content-Type': 'application/json',
    'Accept-Language': 'en',
}

url = "https://signbank.cls.ru.nl/dictionary/api_update_gloss/5/45834/"

senses_engels = [["hire"], ["rent, lease"]]
senses = [["huren"], ["huuuuren"]]

data = {}
data["Senses"] = {}
data["Senses"]["en"] = json.dumps(senses_engels)
data["Senses"]["nl"] = json.dumps(senses) 

print("Data to be sent:")
print(data)

# # Sending the data as raw JSON
response = requests.post(url, headers=headers, json=data)

print(f"Response Status Code: {response.status_code}")

if response.status_code == 200:
    try:
        # Parse JSON response
        json_response = response.json()
        print("JSON Response:")
        print(json.dumps(json_response, indent=4))
    except json.JSONDecodeError:
        print("Failed to decode JSON response.")
elif response.status_code == 500:
    # Save response.text to an HTML file
    with open("error_response.html", "w") as file:
        file.write(response.text)
    print("HTML error response saved to 'error_response.html'.")

    # Parse the HTML response with BeautifulSoup
    soup = BeautifulSoup(response.text, 'html.parser')

    # Find all <li> tags with class "frame user"
    li_elements = soup.find_all('li', class_='frame user')

    # Extract and print the content of the <li> elements
    print("Parsed <li> elements with class 'frame user':")
    for li in li_elements:
        text = li.get_text()
        if text.strip() != "":
            print(text.strip())
else:
    print(f"Unhandled status code: {response.status_code}")
    print("Response Text:")
    print(response.text)

[susanodd]

You are using solely Python and Javascript on your side?

I am relying on what is stored in the database, so making use of objects on the processing side.

It worked when the input senses was the same as what was already in the database. (So it did not need to actually update anything.)

However,
It's true there is a huge problem when you actually want to update the senses.

Unfortunately @Jetske isn't here anymore. She wrote the update senses code.

I need to rewrite it entirely since it isn't working at all now. (I'm working on it locally to get it to stop causing constraint violations.)

Unfortunately, neither @Woseseltops nor @vanlummelhuizen worked on the senses code.
@vanlummelhuizen assisted on the signals code. But that is also not the problem. (I removed the code that was causing that to fail. But it still can't create new senses. It keeps getting a constraint violation. I suspect there is some old object that needs to be removed but was not removed.)

[rem0g]

No I am using two different approaches to update Senses via Python (my script) or Javascript per script you provided.

I would like to ask you to reproduce the problem for me and try to debug the function get_gloss_update_human_readable_value_dict as fast as possible. You can use my script I provided in my previous reply.

[susanodd]

I've got it working now!!

I just rewrote the code that updates the senses. The code is kind of sloppy at the moment.

I'll put in on the branch, then you can see it there. #1463

The code is on signbank-test

Normally, we need to review the code. Since it's very sloppy looking, I think @Woseseltops ought to review it.

You can try it out there to make sure it works. I only used it locally with oodles of print statements.

The updates also show up in the gloss and the revision history.

[vanlummelhuizen]

Did anyone carefully read what @Jetske wrote (which she didn't have to since she is not working for the RU anymore (I hope you had a nice vacation 🌞 ))? See #1360 (comment)

She says that, although not the best implementation, the dict with the senses should be send as a string:

{"Senses": "<a dict serialized as string>"}

That dict-string is than later interpreted here:

Global-signbank/signbank/gloss_update.py

Lines 147 to 170 in 372ef1b

	def convert_string_to_dict_of_list_of_lists(input_string):
	"""
	Convert a string to a dictionary of lists of lists.
	"""
	try:
	dict_of_lists = ast.literal_eval(input_string)
	except (ValueError, SyntaxError):
	return {}, "Sense input is not in the expected format. \nTry: '{\"en\":[[\"sense 1 keyword1\", \"sense 1 keyword2\"],[\"sense 2 keyword1\"]], \"nl\":[[\"sense 1 keyword1\"],[\"sense 2 keyword1\", \"sense 2 keyword2\"]]}'"
	# Verify if the result is a dictionary
	if isinstance(dict_of_lists, dict):
	lengths = set()
	for key, value in dict_of_lists.items():
	if not isinstance(value, list):
	return {}, f"The value associated with key '{key}' is not a list."
	lengths.add(len(value))
	for item in value:
	if not isinstance(item, list):
	return {}, f"The value associated with key '{key}' contains a non-list element."
	if len(lengths) > 1:
	return {}, "Lists within the dictionary have different lengths."
	return dict_of_lists, None
	else:
	return {}, "Input is not a dictionary."

The API docs are confusing. So, the code should probably be improved upon. But this could have been solved by reading things carefully, both code and relevant Github comments.

[susanodd]

It does not work when it's a string. The code is indeed called, but not on the actual "json" string.

@Woseseltops deleted the original API docs where all the examples were. Now there is only Swagger. (There are bugs in the Swagger.)

At the time the comments were written, there was no Swagger. All the video API calls were modified after that so that Swagger would work.

[vanlummelhuizen]

It does not work when it's a string.

There is a bug, yes. But it is not because it is a string. It is this:

diff --git a/signbank/gloss_update.py b/signbank/gloss_update.py
index fc082ff7..1db25a31 100755
--- a/signbank/gloss_update.py
+++ b/signbank/gloss_update.py
@@ -478,7 +478,7 @@ def gloss_update_do_changes(user, gloss, changes, language_code):
                 original_senses = collect_revision_history_for_senses(gloss)
                 update_senses(gloss, new_value)
                 new_senses = collect_revision_history_for_senses(gloss)
-                changes_done.append((field.name, original_senses, new_senses))
+                changes_done.append((field, original_senses, new_senses))
             elif field.name == 'semField':
                 update_semantic_field(gloss, new_value, language_code)
                 changes_done.append((field.name, original_value, new_value))

My API test:

> curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer <my-secret-key>" \
    -d "{\"Senses\": \"{'en': [['hire'], ['rent, lease']], 'nl': [['huren'], ['huren']]}\"}" \
    localhost:8001/dictionary/api_update_gloss/5/47662/
{"glossid": "47662", "errors": {}, "updatestatus": "Success"}

Note that "{\"Senses\": \"{'en': [['hire'], ['rent, lease']], 'nl': [['huren'], ['huren']]}\"}" in the command above is actually the JSON {"Senses": "{'en': [['hire'], ['rent, lease']], 'nl': [['huren'], ['huren']]}"}, where {'en': [['hire'], ['rent, lease']], 'nl': [['huren'], ['huren']]} is in double quotes and therefore a string.

[susanodd]

It does not work when it's a string.

There is a bug, yes. But it is not because it is a string. It is this:

diff --git a/signbank/gloss_update.py b/signbank/gloss_update.py
index fc082ff7..1db25a31 100755
--- a/signbank/gloss_update.py
+++ b/signbank/gloss_update.py
@@ -478,7 +478,7 @@ def gloss_update_do_changes(user, gloss, changes, language_code):
original_senses = collect_revision_history_for_senses(gloss)
update_senses(gloss, new_value)
new_senses = collect_revision_history_for_senses(gloss)

•            changes_done.append((field.name, original_senses, new_senses))
  

•            changes_done.append((field, original_senses, new_senses))
         elif field.name == 'semField':
             update_semantic_field(gloss, new_value, language_code)
             changes_done.append((field.name, original_value, new_value))
  

My API test:

curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer "
-d "{"Senses": "{'en': [['hire'], ['rent, lease']], 'nl': [['huren'], ['huren']]}"}"
localhost:8001/dictionary/api_update_gloss/5/47662/
{"glossid": "47662", "errors": {}, "updatestatus": "Success"}
Note that "{\"Senses\": \"{'en': [['hire'], ['rent, lease']], 'nl': [['huren'], ['huren']]}\"}" in the command above is actually the JSON {"Senses": "{'en': [['hire'], ['rent, lease']], 'nl': [['huren'], ['huren']]}"}, where {'en': [['hire'], ['rent, lease']], 'nl': [['huren'], ['huren']]} is in double quotes and therefore a string.

Yes, I already fixed that in the pull request.
The bug happened before that. During the "update_senses(gloss, new_value)"
It gives transaction management /consistency/unique constraint violated. That is the problem. It is during the "reorder_senses".

This did not used to happen because nobody was updating the senses. They have been using Batch Edit on Signbank.
The code has existed since October. Only now did somebody report an error.

[susanodd]

Senses update doesnt work anymore again:

Payload:

{
  "Senses": {
    "en": [
      [
        "string"
      ]
    ],
    "nl": [
      [
        "string"
      ]
    ]
  }
}

AttributeError at /dictionary/api_update_gloss/5/47662/

'dict' object has no attribute 'strip'

@vanlummelhuizen this is the bug that reopened this issue. It's about a string. There was already a pull request and merge for the "parsing" part of the input.

But then another bug happened. If the senses field had changed from the stored senses, then the "update" caused constraint violations due to the transactions being duplicated. That was caused by the "reorder_senses" method that was being called in a signal.

[Woseseltops]

Okay, we have a quick solution (provided by @vanlummelhuizen ) and a larger solution (proposed by @susanodd) in PR #1463 . The quick solution is already live in production. I have tested both using the curl command suggested in this thread, in both cases providing the senses as string as indicated by Jetske.

Manual test results:

• Quick solution (tested on the production server): the error goes away, but it only deletes existing senses without adding new ones.
• PR (tested on signbank-test): I'm seeing the senses as intended.

Seems like we need to review PR #1463 as quickly as possible; to be continued there.

[susanodd]

Okay, we have a quick solution (provided by @vanlummelhuizen ) and a larger solution (proposed by @susanodd) in PR #1463 . The quick solution is already live in production. I have tested both using the curl command suggested in this thread, in both cases providing the senses as string as indicated by Jetske.

Manual test results:

• Quick solution (tested on the production server): the error goes away, but it only deletes existing senses without adding new ones.

URGENT disable updating senses in the API, before the API accidentally deletes senses