Error while importing cha annotations

[Valentina-Fed]

Hi,

I get an error when I try to import and index cha annotations: child-project import-annotations . --set cha --recording_filename ZQM_040622.mp3 --time_seek 0 --range_onset 0 --range_offset 2264128 --raw_filename ZQM_040622.cha --format cha

It occurs during processing the annotation:
warning: some of the converters do not support multithread importation; running on 1 thread
an error occured while processing './annotations/cha/raw/ZQM_040622.cha'

And then: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
KeyError: ['annotation_filename']

Thank you!

[lucasgautheron]

Hello!

Can you post the full stracktrace here? Just to know which part of the code is throwing this error.

[Valentina-Fed]

Sorry, one more error:

an error occured while processing './annotations/cha/raw/LMC_D14-14.cha'
Traceback (most recent call last):
File "/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/ChildProject/annotations.py", line 430, in _import_annotation
df = converter.convert(path, filter)
File "/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/ChildProject/converters.py", line 610, in convert
lambda r: r["tiers"][r["participant"]], axis=1
File "/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/pandas/core/generic.py", line 5487, in getattr
return object.getattribute(self, name)
AttributeError: 'DataFrame' object has no attribute 'str'

/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/pandas/core/frame.py:5182: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
downcast=downcast,
Traceback (most recent call last):
File "/home/vfedchenko/.conda/envs/childproject/bin/child-project", line 8, in
sys.exit(main())
File "/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/ChildProject/cmdline.py", line 518, in main
args.func(args)
File "/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/ChildProject/cmdline.py", line 195, in import_annotations
errors, warnings = am.validate(annotations=imported, threads=args.threads)
File "/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/ChildProject/annotations.py", line 360, in validate
annotations = annotations.dropna(subset=["annotation_filename"])
File "/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/home/vfedchenko/.conda/envs/childproject/lib/python3.7/site-packages/pandas/core/frame.py", line 5953, in dropna
raise KeyError(list(np.compress(check, subset)))
KeyError: ['annotation_filename']

It occurs only with the files that are entitled like this LMC_D14-14.cha (original title from the corpus).

[lucasgautheron]

These files look empty:

@UTF8
@PID:	11312/a-00016616-1
@Begin
@Languages:	nan
@Participants:	CHI Target_Child
@ID:	nan|Tsay|CHI|||||Target_Child|||
@Media:	D14-14, audio, notrans
@Comment:	This is a dummy file to permit playback from the TalkBank
	browser.  Please use the slider at the left to control media
	playback.
@End

I think they should not be imported.
I will update the package to make the error message more meaningful for such cases.

[Valentina-Fed]

Yes, you are right, but we still would like to use these recordings, so we should think what to do with empty annotations. @alecristia
I've seen also that some of converted annotations had empty csv-files, but their cha-equivalents are not empty. For the moment, I can not understand, why some of annotations were well converted and some not. For example, all the ZQM, LMC and WZX annotations are empty.

[lucasgautheron]

Yes, you are right, but we still would like to use these recordings, so we should think what to do with empty annotations.
You can use these recordings still (and include them in the dataset) - they will just not be covered by this set of annotations.

@alecristia I've seen also that some of converted annotations had empty csv-files, but their cha-equivalents are not empty. For the moment, I can not understand, why some of annotations were well converted and some not. For example, all the ZQM, LMC and WZX annotations are empty.

This is a problem though. I think this might require some tweaking of the ChatConverter. Can you give an example of a chat that is imported without trouble and one that comes empty? I'd need to have a look at why pylangacq seems to handle those differently.

It would be useful to upload your work on GIN so I can have a look and do some testing :)

[lucasgautheron]

Actually I think I know why those annotations are empty. It is because the package drops segments that have no timestamp (https://github.com/LAAC-LSCP/ChildProject/blob/master/ChildProject/converters.py#L654). It seems that ZQM transcriptions are not timestamped. I am not sure what we should do in that case @alecristia

[alecristia]

yes, they should be dropped when there is no time stamp, so the behavior of the package is correct, but perhaps we should have a more informative warning message --

…

--------------------------------------------------------------- Alex (Alejandrina) Cristia Researcher, CNRS Laboratoire de Sciences Cognitives et Psycholinguistique 29, rue d'Ulm, 75005, Paris, FRANCE My site: www.acristia.org --------------------------------------------------------------- If you donate, ask me about effective charities <https://effectivealtruism.us8.list-manage.com/track/click?u=52b028e7f799cca137ef74763&id=206509456c&e=6c0b626a8f>. / Si vous faites des dons, demandez moi sur le don efficace <https://www.altruismeefficacefrance.org/guide-don-efficace-1/>.

[Valentina-Fed]

In fact, there is no warning about empty csv. They are generated as usual, but when you look inside, there is only one line of headers.