[rocketchat] Add support for RocketChat

[animeshk08]

This commit adds support for Pagure
to elk. Raw and enriched indexes have
been added.

Fixes: #876

Signed-off-by: Animesh Kumar [email protected]

[animeshk08]

Hi, @valeriocos I sincerely apologize for the delay with this work. I was really occupied for the past few weeks.

I tried to include all the important field coming from the backend, let me know if anything is missed. I will include the remaining files and tests once we finalize on the enricher.

Again, I apologize and thank you for your patience.

\cc @vchrombie

EDIT: The format to pass to Mordred is same as mentioned here: chaoss/grimoirelab-perceval#661 (review) and the testing can be done on these channels,https://open.rocket.chat/channel/sandbox and https://open.rocket.chat/channel/general or a custom channel(suggested)
EDIT2: Tests are failing as Perceval backend is not merged

[coveralls]

Pull Request Test Coverage Report for Build 2210

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 36 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.7%) to 81.246%

---

Files with Coverage Reduction	New Missed Lines	%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/utils.py	36	65.78%

Totals
Change from base Build 2208:	-0.7%
Covered Lines:	8140
Relevant Lines:	10019

---

💛 - Coveralls

[valeriocos]

Hi @animeshk08 , sorry for the late reply! Lately I don't have too much time to dedicate to community issues/pull requests, so I do it when I get some time.

The PR looks great, I tested it on the whole "https://open.rocket.chat general" and the execution was successful. I left some comments.

Based on the implementation you proposed, can you list some metrics you would like to show in the dashboard? Do you think we can be close to the metrics for other messaging systems (e.g., slack, mattermost)?

[valeriocos]

This field seems to be always empty. Do you have an example?
Does it make sense to store only the number of replies?

[valeriocos]

The current implementation generates a mapping for any reaction type (https://gist.github.com/valeriocos/9b103650bc24df8185aecb8f5df3a43f#file-rocketchat-mapping-L156), this may be a problem when trying to summing up reactions of different types.

I would suggest to:

• add a metric that shows the total number of reactions for a given message
• modify the code to have a single mapping for reactions, which contains a list of names, usernames, the type of the reaction and the count of reactions for that type (basically the lenght of the list of names/usernames)

"reaction" : {
            "properties" : {
              "names" : {
                "type" : "keyword"
              },
              "usernames" : {
                "type" : "keyword"
              },
               "count" : {
                "type" : "long"
              },
               "type" : {
                "type" : "keyword"
              }
            }
          },

WDYT?

[valeriocos]

I would complete this method with the total number of mentions, it may be useful to calculate some global metrics (e.g., number of mentions in different channels), wdyt?

[valeriocos]

why do we store the lastMessage info?

[valeriocos]

Why we want to store also the meta info? Is the URL not enough?

[valeriocos]

It can be useful to add an attribute that says whether the message has been edited or not (e.g., is_edited = 0/1). This can be useful to count the number of edited messages

[valeriocos]

The grimoire_creation_date isn't present in the enriched_items, you should add a statement similar to: https://github.com/chaoss/grimoirelab-elk/blob/master/grimoire_elk/enriched/mattermost.py#L193

[lukaszgryglicki]

This PR is not working yet, see p2o.py --enrich --index test-rc-raw --index-enrich test-rc -e [redacted] --bulk-size 50 --scroll-size 50 --db-host [redacted] --db-sortinghat sortinghat --db-user sortinghat --db-password [redacted] rocketchat -t [redacted] -u [redacted] --from-date '2020-06-04' https://chat.hyperledger.org general:

Traceback (most recent call last):
  File "/repos/grimoirelab-elk/grimoire_elk/elk.py", line 530, in enrich_backend
    enrich_count = enrich_items(ocean_backend, enrich_backend)
  File "/repos/grimoirelab-elk/grimoire_elk/elk.py", line 318, in enrich_items
    total = enrich_backend.enrich_items(ocean_backend)
  File "/repos/grimoirelab-elk/grimoire_elk/enriched/enrich.py", line 389, in enrich_items
    rich_item = self.get_rich_item(item)
  File "/repos/grimoirelab-elk/grimoire_elk/enriched/enrich.py", line 93, in decorator
    eitem = func(self, *args, **kwargs)
  File "/repos/grimoirelab-elk/grimoire_elk/enriched/rocketchat.py", line 133, in get_rich_item
    eitem['mentions'] = self.__get_mentions(message['mentions'])
  File "/repos/grimoirelab-elk/grimoire_elk/enriched/rocketchat.py", line 170, in __get_mentions
    'name': usr['name']})
KeyError: 'name'

This fixes the problem vim /repos/grimoirelab-elk/grimoire_elk/enriched/rocketchat.py:

         for usr in mentioned:
-            if '_id' in usr.keys():
+            if '_id' in usr.keys() and 'name' in usr.keys():

[valeriocos]

@animeshk08 do you have time to finish this PR or @obaroikoh can take over?

[animeshk08]

Hi @valeriocos I am a bit occupied for the coming weeks. Thank you for the patch @obaroikoh you can take over this PR.

[valeriocos]

Thank you for the quick reply @animeshk08