[rocketchat] Rocketchat patch for PR #882 #890

obaroikoh · 2020-06-10T12:53:04Z

Hi Team,

We needed to use this PR hence this patch.

It adds grimoire_creation_date and is_edited to the enriched date and this

Cheers

Sample output

        "_index": "testing-rc",
        "_type": "_doc",
        "_id": "14942d6b2993eabcbcbb9d62cf1008b4a5fa96bb",
        "_score": 1,
        "_source": {
          "author_domain": null,
          "repository_labels": null,
          "author_name": "wangzd",
          "channel_id": "GENERAL",
          "metadata__enriched_on": "2020-06-10T12:34:18.850454+00:00",
          "author_org_name": "Unknown",
          "author_user_name": "wangzd",
          "grimoire_creation_date": "2020-06-01T09:31:16.639000+00:00",
          "user_id": "pvxYbDt8iwnJ4dQMT",
          "u_gender_acc": 0,
          "author_id": "7126db4494ac080d6c371bdf0cc34e85f8863732",
          "user_username": "wangzd",
          "metadata__gelk_version": "0.72.0",
          "author_multi_org_names": [
            "Unknown"
          ],
          "u_gender": "Unknown",
          "author_uuid": "7126db4494ac080d6c371bdf0cc34e85f8863732",
          "metadata__gelk_backend_name": "RocketChatEnrich",
          "metadata__updated_on": "2020-06-01T09:31:16.639000+00:00",
          "origin": "https://chat.hyperledger.org/general",
          "u_bot": false,
          "u_user_name": "wangzd",
          "rid": "GENERAL",
          "metadata__timestamp": "2020-06-10T12:31:22.334395+00:00",
          "u_uuid": "7126db4494ac080d6c371bdf0cc34e85f8863732",
          "author_gender": "Unknown",
          "user_name": null,
          "msg": "wangzd",
          "author_gender_acc": 0,
          "author_bot": false,
          "metadata__filter_raw": null,
          "uuid": "14942d6b2993eabcbcbb9d62cf1008b4a5fa96bb",
          "channel_num_users": 24996,
          "text_analyzed": "wangzd",
          "is_edited": 0,
          "u_id": "7126db4494ac080d6c371bdf0cc34e85f8863732",
          "channel_name": "general",
          "is_rocketchat_message": 1,
          "u_org_name": "Unknown",
          "id": "3prnDwX5vm6iLp6SB",
          "u_name": "wangzd",
          "parent": null,
          "channel_num_messages": 43882,
          "u_domain": null,
          "tag": "https://chat.hyperledger.org/general",
          "channel_updated_at": "2020-06-10T11:39:38.847000+00:00",
          "offset": null,
          "u_multi_org_names": [
            "Unknown"
          ]
        }

coveralls · 2020-06-10T13:13:30Z

Pull Request Test Coverage Report for Build 2289

0 of 0 changed or added relevant lines in 0 files are covered.
35 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.08%) to 81.96%

Files with Coverage Reduction	New Missed Lines	%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/utils.py	35	66.16%

Totals
Change from base Build 2281:	0.08%
Covered Lines:	8255
Relevant Lines:	10072

💛 - Coveralls

valeriocos

Hi @obaroikoh ,

Please check the comments, and add the schema and tests. You can have a look the PR #881 as a reference.

Let me know if you need any help.

Thank you

grimoire_elk/enriched/rocketchat.py

obaroikoh · 2020-06-11T10:22:38Z

Hi @valeriocos, I have added some tests, the schema and sample raw data. I also looked at the comments in the other other PR and made some of the changes you recommended. However, I didn't handle the concern you raised wrt storing lastMessage and url meta info because I wasn't sure why we are doing it. Perheps @animeshk08 can help give more context.

animeshk08 · 2020-06-11T10:26:53Z

Hi @obaroikoh Thank you for the updates! Great work! I had added those two fields as I found it to be unique for this backend. However, they do not provide any necessary insights hence you may skip them for the first version of this backend.Thanks!

obaroikoh · 2020-06-11T10:40:46Z

@animeshk08 Thanks for the quick response. I will skip them as you've suggested.

valeriocos · 2020-06-12T09:49:35Z

Hi @obaroikoh , can I review the PR?

obaroikoh · 2020-06-12T11:37:37Z

Sure you can. I was comparing the enriched data with slack and noticed some fields were missing, avatar and channel_topic and had to update the PR

valeriocos · 2020-06-12T15:07:44Z

Hi @obaroikoh , I'm starting reviewing the PR (checking the code and running the code on some rocketchat channels)

EDIT: I'm performing a long run, tomorrow morning it should be over.

valeriocos

Hi @obaroikoh, sorry for the delay. Overall the PR looks fine, I left some comments (and suggestions), please check them when you have time.

I have tested the code on a channel with more than 150.000 messages, the collection and enrichment worked fine. The method __get_reactions should be modified to remove the attribute "properties" from the dictionaries generated.

Please run the test coverage (ref here) to make sure that all code is covered (if possible).

Thank you for working on this PR.

valeriocos · 2020-06-13T11:55:36Z

grimoire_elk/enriched/rocketchat.py

+        mapping = """
+        {
+            "properties": {
+                "text_analyzed": {


please rename test_analyzed with msg_analyzed

Hi @obaroikoh, sorry for the delay. Overall the PR looks fine, I left some comments (and suggestions), please check them when you have time.

I have tested the code on a channel with more than 150.000 messages, the collection and enrichment worked fine. The method __get_reactions should be modified to remove the attribute "properties" from the dictionaries generated.

Please run the test coverage (ref here) to make sure that all code is covered (if possible).

Thank you for working on this PR.

Thanks for the review. I will make all necessary changes, hopefully before the end of the weekend.

You're welcome! take your time :)

I have made the changes but while testing enrichment against my dev environment, I got this error

File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 584, in _build_master ws.require(__requires__) File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 901, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 792, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.VersionConflict: (grimoire-elk 0.63.0 (/usr/local/lib/python3.5/dist-packages), Requirement.parse('grimoire-elk==0.72.0')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/bin/p2o.py", line 4, in <module> __import__('pkg_resources').require('grimoire-elk==0.72.0') File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 3261, in <module> @_call_aside File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 3245, in _call_aside f(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 3274, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 586, in _build_master return cls._build_from_requirements(__requires__) File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements dists = ws.resolve(reqs, Environment()) File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 792, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (numpy 1.18.5 (/usr/local/lib/python3.5/dist-packages), Requirement.parse('numpy<=1.18.3'), {'cereslib'})

command:

p2o.py --enrich --index testing-rc-raw --index-enrich testing-rc -e ES-URL --bulk-size 500 --scroll-size 500 --db-host REDACTED --db-sortinghat REDACTED --db-user REDACTED --db-password REDACTED rocketchat -t REDACTED -u REDACTED --from-date '2020-06-10' https://chat.hyperledger.org general

can you try to constrain the dep of numpy (#891) and test it again?

okay. let me try that

It worked. Thanks

grimoire_elk/enriched/rocketchat.py

valeriocos · 2020-06-13T12:15:18Z

grimoire_elk/enriched/rocketchat.py

+
+        for usr in mentioned:
+            if '_id' in usr.keys() and 'name' in usr.keys():
+                rich_mentions.append({'username': usr['username'],


The if condition can be replaced by accessing the dict's attributes with get, please save the dict in a variable (this will ease future debugging operations).

for usr in mentioned: - if '_id' in usr.keys() and 'name' in usr.keys(): - rich_mentions.append({'username': usr['username'], - 'id': usr['_id'], - 'name': usr['name']}) + rich_mention = { + 'username': usr.get('username', None), + 'id': usr.get('_id', None), + 'name': usr.get('name', None) + } + rich_mentions.append(rich_mention)

grimoire_elk/enriched/rocketchat.py

valeriocos · 2020-06-13T12:17:38Z

schema/rocketchat.csv

@@ -0,0 +1,50 @@
+name,type,aggregatable,description


Please update the CSV by adding the new attributes and removing the old ones, thanks

valeriocos · 2020-06-16T07:15:42Z

Hi @obaroikoh , can I review the PR?

obaroikoh · 2020-06-16T07:38:34Z

Sure you can. Thank you.

valeriocos

Hi @obaroikoh , thank you for addressing my previous comments!

I have tested the PR and works fine. I left some comments concerning the CSV, which doesn't include some attributes. After addressing this last round of comments, I think the PR can merged.

Thank you for your work and time :)

schema/rocketchat.csv

This commit adds support for Pagure to elk. Raw and enriched indexes have been added. Signed-off-by: Animesh Kumar <[email protected]>

obaroikoh · 2020-06-17T11:54:17Z

Hi @obaroikoh , thank you for addressing my previous comments!

I have tested the PR and works fine. I left some comments concerning the CSV, which doesn't include some attributes. After addressing this last round of comments, I think the PR can merged.

Thank you for your work and time :)

Hi @valeriocos, once again thanks for the review. I have made the changes in the comments. You can review it when you have the time.

Cheers.

valeriocos · 2020-06-18T09:13:18Z

Hi @obaroikoh , thank you for your time and patience. I have just reviewed the PR, it looks great.

I have submitted a PR to your fork obaroikoh#1 to increase the test coverage and complete the documentation.

obaroikoh · 2020-06-18T09:59:51Z

Hi @valeriocos, Thanks once again for the review. While doing further testing, we found that in rare cases where topic isn't set under channel_info, enrichment breaks. I have fixed that and will be updating my PR.

Signed-off-by: Obaro Ikoh <[email protected]>

valeriocos · 2020-06-18T10:53:07Z

Ok, perfect @obaroikoh ! Can you review the PR obaroikoh#1 (or include the changes in this PR)?

Once done, this PR is ready to merge.

Please feel free to review also chaoss/grimoirelab-sirmordred#477, it adds the instructions to execute rocketchat from Mordred.

Final question :) are you interested in creating a dashboard to visualize the data extracted from Rocketchat? If this is the case, a good start would be to mimic the dashboard for Slack (https://github.com/chaoss/grimoirelab-sigils/blob/master/json/slack.json).

obaroikoh · 2020-06-18T11:16:00Z

Ok, perfect @obaroikoh ! Can you review the PR obaroikoh#1 (or include the changes in this PR)?

I have reviewed this. I saw a line where I was adding the avatar and noticed I wasn't using the python dict get method, on line 210. Everything else looks good.

Final question :) are you interested in creating a dashboard to visualize the data extracted from Rocketchat? If this is the case, a good start would be to mimic the dashboard for Slack (https://github.com/chaoss/grimoirelab-sigils/blob/master/json/slack.json).

I have created one already :)

Please feel free to review also chaoss/grimoirelab-sirmordred#477, it adds the instructions to execute rocketchat from Mordred.

Sure, I will do that.

valeriocos · 2020-06-18T11:28:52Z

I have reviewed this. I saw a line where I was adding the avatar and noticed I wasn't using the python dict get method, on line 210. Everything else looks good.

Ok, perfect! You should merge the PR obaroikoh#1 and then update this PR (to include the commit obaroikoh@2a1fdee). If for some reason this isn't possible, you can copy the content of the commit above to this PR and close obaroikoh#1 without merging.

I have created one already :)

That's great! Would you like to contribute to Sigils by submitting a PR with the dashboard you created?

Sure, I will do that.

Thanks

valeriocos

LGTM, thanks @obaroikoh

obaroikoh · 2020-06-19T10:06:51Z

Hi @valeriocos, Apologies for the delay. I see that you have merged the PR. Thanks.

Many thanks to @animeshk08.

Ok, perfect! You should merge the PR obaroikoh#1 and then update this PR (to include the commit obaroikoh@2a1fdee). If for some reason this isn't possible, you can copy the content of the commit above to this PR and close obaroikoh#1 without merging.

Was this taken care of before the merge?

That's great! Would you like to contribute to Sigils by submitting a PR with the dashboard you created?

Sure. I'd submit one this weekend.

valeriocos · 2020-06-19T10:09:37Z

No worries @obaroikoh ! Since we are preparing a new release I thought that it was good to include this contribution in it.

obaroikoh · 2020-06-19T10:12:19Z

Okay great.

animeshk08 · 2020-06-19T10:25:57Z

Great work! @obaroikoh

valeriocos reviewed Jun 10, 2020

View reviewed changes

grimoire_elk/enriched/rocketchat.py Show resolved Hide resolved

grimoire_elk/enriched/rocketchat.py Outdated Show resolved Hide resolved

obaroikoh force-pushed the rocket-chat-patch branch 2 times, most recently from 4106d06 to 2db3af2 Compare June 11, 2020 09:45

obaroikoh force-pushed the rocket-chat-patch branch 2 times, most recently from 96ded00 to af2ea67 Compare June 12, 2020 07:53

obaroikoh force-pushed the rocket-chat-patch branch from af2ea67 to 8452e88 Compare June 12, 2020 11:33

obaroikoh force-pushed the rocket-chat-patch branch 2 times, most recently from 48c080a to 1b728d4 Compare June 12, 2020 13:51

valeriocos reviewed Jun 13, 2020

View reviewed changes

obaroikoh force-pushed the rocket-chat-patch branch from 1b728d4 to 63274d1 Compare June 13, 2020 18:45

valeriocos reviewed Jun 16, 2020

View reviewed changes

[rocketchat] Add support for RocketChat

5ed7c54

This commit adds support for Pagure to elk. Raw and enriched indexes have been added. Signed-off-by: Animesh Kumar <[email protected]>

obaroikoh force-pushed the rocket-chat-patch branch from 63274d1 to 4b2b76c Compare June 17, 2020 11:30

valeriocos mentioned this pull request Jun 18, 2020

[doc] Add instructions to mine rocketchat chaoss/grimoirelab-sirmordred#477

Merged

[rocketchat] Patch for PR chaoss#882

b73582a

Signed-off-by: Obaro Ikoh <[email protected]>

obaroikoh force-pushed the rocket-chat-patch branch from 4b2b76c to b73582a Compare June 18, 2020 10:02

valeriocos self-requested a review June 19, 2020 09:57

valeriocos approved these changes Jun 19, 2020

View reviewed changes

valeriocos merged commit be6ae30 into chaoss:master Jun 19, 2020

vchrombie mentioned this pull request Jun 19, 2020

[Rocket.Chat] Add rocketchat backend support to ELK #876

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rocketchat] Rocketchat patch for PR #882 #890

[rocketchat] Rocketchat patch for PR #882 #890

obaroikoh commented Jun 10, 2020

coveralls commented Jun 10, 2020 •

edited

Loading

valeriocos left a comment

obaroikoh commented Jun 11, 2020 •

edited

Loading

animeshk08 commented Jun 11, 2020

obaroikoh commented Jun 11, 2020

valeriocos commented Jun 12, 2020

obaroikoh commented Jun 12, 2020

valeriocos commented Jun 12, 2020 •

edited

Loading

valeriocos left a comment

valeriocos Jun 13, 2020

obaroikoh Jun 13, 2020

valeriocos Jun 13, 2020

obaroikoh Jun 13, 2020 •

edited

Loading

valeriocos Jun 13, 2020

obaroikoh Jun 13, 2020

obaroikoh Jun 13, 2020

valeriocos Jun 13, 2020

valeriocos Jun 13, 2020

valeriocos commented Jun 16, 2020

obaroikoh commented Jun 16, 2020

valeriocos left a comment

obaroikoh commented Jun 17, 2020

valeriocos commented Jun 18, 2020

obaroikoh commented Jun 18, 2020

valeriocos commented Jun 18, 2020

obaroikoh commented Jun 18, 2020

valeriocos commented Jun 18, 2020

valeriocos left a comment

obaroikoh commented Jun 19, 2020

valeriocos commented Jun 19, 2020

obaroikoh commented Jun 19, 2020

animeshk08 commented Jun 19, 2020

[rocketchat] Rocketchat patch for PR #882 #890

[rocketchat] Rocketchat patch for PR #882 #890

Conversation

obaroikoh commented Jun 10, 2020

coveralls commented Jun 10, 2020 • edited Loading

Pull Request Test Coverage Report for Build 2289

💛 - Coveralls

valeriocos left a comment

Choose a reason for hiding this comment

obaroikoh commented Jun 11, 2020 • edited Loading

animeshk08 commented Jun 11, 2020

obaroikoh commented Jun 11, 2020

valeriocos commented Jun 12, 2020

obaroikoh commented Jun 12, 2020

valeriocos commented Jun 12, 2020 • edited Loading

valeriocos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

obaroikoh Jun 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valeriocos commented Jun 16, 2020

obaroikoh commented Jun 16, 2020

valeriocos left a comment

Choose a reason for hiding this comment

obaroikoh commented Jun 17, 2020

valeriocos commented Jun 18, 2020

obaroikoh commented Jun 18, 2020

valeriocos commented Jun 18, 2020

obaroikoh commented Jun 18, 2020

valeriocos commented Jun 18, 2020

valeriocos left a comment

Choose a reason for hiding this comment

obaroikoh commented Jun 19, 2020

valeriocos commented Jun 19, 2020

obaroikoh commented Jun 19, 2020

animeshk08 commented Jun 19, 2020

coveralls commented Jun 10, 2020 •

edited

Loading

obaroikoh commented Jun 11, 2020 •

edited

Loading

valeriocos commented Jun 12, 2020 •

edited

Loading

obaroikoh Jun 13, 2020 •

edited

Loading