opening a tagged Persian Corpus in Inception

[NiloofarTUDarmstadt]

Hi,
I have opened a Part of Speech tagged Corpus as plain text in inception and I am going to annotate the text as another layer.

1. How can inception recognize the tags which are already in the text and make it as a layer?
2. I have attached the screenshot how inception opened the text, can someone tell me what is the problem and why the text has spaces there?
3. Maybe i should import the text in another format, can anybody help me about it?
   great thanks before,
   Niloofar
   

[reckart]

It does not look like a familar format to me. Is that corpus maybe available in CoNLL-U format?

[NiloofarTUDarmstadt]

Unfortunately it is a really restricted resource.
How I should know in which format it already is?
The main corpus is Bijankhan Persian corpus

[reckart]

Here you can find an overview of the currently supported formats in INCEpTION - most of them come with an example:

https://inception-project.github.io/releases/32.3/docs/user-guide.html#sect_formats

If your format matches any of these, select the respective format during import.

If none of the formats matches your corpus, you need to convert it first - e.g. by implementing a suitable Python script.

[NiloofarTUDarmstadt]

The provided POS annotated corpus follows a tab-delimited format, where each line contains a word followed by its corresponding part of speech (POS) tag, separated by a tab. Additionally, sentences are separated by lines containing only the "#" character.

Could you please suggest me which format i should use?
new.txt

I want that inception recognizes the (POS) tags so that on a different layer I start annotating the corpus according to discourse aspect of modality.

[reckart]

Such a format is not supported out-of-the-box by INCEpTION. The closest ones could be

• The CoNLL 2000 format represents POS and Chunk tags. Fields in a line are separated by spaces. Sentences are separated by a blank new line.
• The CoNLL 2006 (aka CoNLL-X) format targets dependency parsing. Columns are tab-separated. Sentences are separated by a blank new line.

Can you convert your data to either of these?

[NiloofarTUDarmstadt]

I work on it in Pzthon or Java. Thank you so much! I will write Feedback here.

[NiloofarTUDarmstadt]

Hi again!

1. The Persian text is tokenized and annotated with POS. The recommended format you suggest for me is still CoNLL 2000, although I do not have chunk tags? I will need only the layer of POS tags finally + my own layer of Conversational Background to compare. Because I am trying to find the code to change the format of the annotated corpus to the possible readable format in INCEPTION.

2. After I have changed the format, should I add the tagset to the layer or it realizes the Tagset automatically?

Thank you for always being supportive.

[NiloofarTUDarmstadt]

I do not work on (Named Entity Recognition) and only on POS tags layer and my own layer

[NiloofarTUDarmstadt]

the code provided here:
https://colab.research.google.com/github/inception-project/inception/blob/master/notebooks/using_pretokenized_and_preannotated_text.ipynb#scrollTo=rkpxrzHZXW-L
identifies the NamedEntity Tags and not the POS tags...
Could you help me with a code ...

[reckart]

The code there illustrates how to convert pre-annotated data into XMI format. If you go that route, it is necessary to observe a special convention: it is not only necessary to create the POS annotations of type de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS but also to attach each of these POS annotations to the pos feature of corresponding Token annotation (de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token). So you would need to create three levels of annotations:

• de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Sentence
• de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token
• de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.POS
• write your POS tags to the PosValue feature of the de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.POS annotations
• connect your POS annotations to the pos feature of the de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token annotations

Instead though, you could transform your data to the CoNLL-2006 format which is

1	go	walk	VERB	_	_	_	_	_

and then import that into INCEpTION.