MG-2142 - Consume Things connect/disconnect event in bootstrap #2192

JeffMboya · 2024-04-19T12:46:39Z

What type of PR is this?

This is a feature because it adds the connectEvent struct to the consumer package.

What does this do?

If channel and thing are connected , change state of config to Active
If channel and thing are disconnected, change state of config to Inactive

Which issue(s) does this PR fix/relate to?

Resolves Consume Things connect event in Bootstrap #2142

Have you included tests for your changes?

Yes, I have included tests for my changes.

Did you document any new/modified feature?

No

bootstrap/postgres/configs.go

arvindh123

I have question here, The event published by connectThingEvent and disconnectThingEvent are same event published by thing service on disconnect and connect ?

JeffMboya · 2024-04-26T10:08:34Z

I have question here, The event published by connectThingEvent and disconnectThingEvent are same event published by thing service on disconnect and connect ?

They are not the same. The event published by things service is a notification that a thing has been connected or disconnected from a channel. The event published by connectThingEvent and disconnectThingEvent in bootstrap are an acknowledgement that a thing has been connected or disconnected from a channel

bootstrap/postgres/configs.go

arvindh123

@JeffMboya I have suggestion for handling error rollbacks

bootstrap/events/consumer/streams.go

arvindh123

While trying to assign parent group for channel , like below request

curl --location 'http://localhost/channels/795b0be8-f8d9-4101-a40f-859ed573c71c/groups/assign' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <Token>' \
--data '{
    "group_ids" : [ "5c6de392-6dc4-46fc-8e2c-e6f4ce4fffab" ]
}'

In events received below logs, which use group.assign operation , the same operation is used for thing connect, so could not different between thing connection event and group assign channel events

[#8] Received on "events.magistrala.things" with reply "_INBOX.7g7jymtBXH0wOSok79kUGo.0LlN1Z1P"
*�{"group_id":"795b0be8-f8d9-4101-a40f-859ed573c71c","member_ids":["5c6de392-6dc4-46fc-8e2c-e6f4ce4fffab"],"occurred_at":1714131210669902557,"operation":"group.assign"}

Additionally,

In the event message group_id filed contains channel id
actually here we should have group id

In member_ids field , it contains
group_id , actually here we should have channel id

This issue might be a blocker for this PR

bootstrap/service.go

rodneyosodo · 2024-05-17T11:44:21Z

bootstrap/postgres/configs_test.go

+	repo := postgres.NewConfigRepository(db, testLog)
+	err := deleteChannels(context.Background(), repo)
+	require.Nil(t, err, "Channels cleanup expected to succeed.")
+


Test this occurrences using table testing:

Connect for an already connected thing

Connect for a disconnected thing

Connect for an invalid thing

Connect for a random thing

Connect for an empty thing

bootstrap/events/producer/events.go

bootstrap/events/consumer/streams.go

bootstrap/configs.go

arvindh123 · 2024-06-21T07:54:46Z

bootstrap/service.go

+	channelsPage, err := bs.sdk.ChannelsByThing(cfg.ThingID, pm, token)
+	if err != nil {
+		return Config{}, errors.Wrap(errCheckChannels, err)
+	}


This will not work , if thing is connect to 100 of channels.
By default the query limit is 10. So with this bs.sdk.ChannelsByThing(cfg.ThingID, pm, token) we could not get all the 100 channels of thing.
There are two options:

Iterate on channelsPage.Total and get all the channels

New API endpoint things to verify the thing and channel are connected. (This option looks for me most efficent way)

arvindh123 · 2024-06-21T07:58:05Z

bootstrap/service.go


+	channelsPage, err := bs.sdk.ChannelsByThing(cfg.ThingID, pm, token)
+	if err != nil {
+		return errors.Wrap(errChangeState, err)
+	}
+
+	for _, channel := range channelsPage.Channels {
+		existingChannels[channel.ID] = true
+	}


I think we don't want this, because sending connect request to already connected channel should not throw error.
If things API throw error. then we should think about changing the response of things API

I think we don't want this, because sending connect request to already connected channel should not throw error. If things API throw error. then we should think about changing the response of things API

@arvindh123 yes, things API throws http status 409 when you send a connect request to an already connected channel.
magistrala-things | {"time":"2024-06-12T10:04:36.568921712Z","level":"WARN","msg":"Assign member to group failed to complete successfully","duration":"67.069682ms","group_id":"8fac693b-dc98-4178-b828-a4d4a8008d05","relation":"group","member_kind":"things","member_ids":["f2097dff-8e03-4461-92f1-80b972bb3c10"],"error":{"error":"entity already exists","message":"failed to add policies"}}

Signed-off-by: JeffMboya <[email protected]>

Signed-off-by: JeffMboya <[email protected]> Add ConnectThing method to bootstrap Signed-off-by: JeffMboya <[email protected]> Consume ThingConnect in bootstrap Signed-off-by: JeffMboya <[email protected]> Consume ThingConnect in bootstrap Signed-off-by: JeffMboya <[email protected]> Consume ThingConnect in bootstrap Signed-off-by: JeffMboya <[email protected]> Consume ThingConnect event Signed-off-by: JeffMboya <[email protected]> Consume ThingsConnect event Signed-off-by: JeffMboya <[email protected]> Consume ThingsConnect event Signed-off-by: JeffMboya <[email protected]> Consume ThingsConnect event Signed-off-by: JeffMboya <[email protected]> Implement ReadStringSlice Signed-off-by: JeffMboya <[email protected]> Add memberKind and relation checks Signed-off-by: JeffMboya <[email protected]> Add memberKind and relation checks Signed-off-by: JeffMboya <[email protected]> Add memberKind and relation checks Signed-off-by: JeffMboya <[email protected]> Add memberKind and relation checks Signed-off-by: JeffMboya <[email protected]> Add memberKind and relation checks Signed-off-by: JeffMboya <[email protected]> Add TestDisconnectThing to configs_test Signed-off-by: JeffMboya <[email protected]> Update mocks Signed-off-by: JeffMboya <[email protected]>

Signed-off-by: JeffMboya <[email protected]> Refactor: revert to thingID and channelID Signed-off-by: JeffMboya <[email protected]>

Signed-off-by: JeffMboya <[email protected]> Refactor: add memberKind ans relation constants Signed-off-by: JeffMboya <[email protected]> Refactor: add memberKind ans relation constants Signed-off-by: JeffMboya <[email protected]> Refactor: add memberKind ans relation constants Signed-off-by: JeffMboya <[email protected]> Refactor: add memberKind ans relation constants Signed-off-by: JeffMboya <[email protected]>

Signed-off-by: JeffMboya <[email protected]>

arvindh123 · 2024-06-21T15:27:40Z

There are some known bug in bootstrap
Feature: An Endpoint to verify thing channel connection #2304
Feature: Return response other than error for connecting existing connection of thing and channel #2303
Bug: In a bootstrap config with multiple channels, bootstrap state goes to 1, if any one of the channel connect to thing absmach/magistrala#12
Bug: Bootstrap config state could not change, where thing and channel connected before config creation absmach/magistrala#11

MG-2142 - Consume Things connect/disconnect event in Bootstrap (absmach#2192) NOISSUE - add CreatedAt and UpdatedAt fields for mgclients.Client (absmach#2306) NOISSUE - Fix failed log message (absmach#2307) MF-2299 - Update influxdb2 volumes path (absmach#2300) NOISSUE - Remove redundant relation check (absmach#2197) NOISSUE - Add domain events for Auth service (absmach#2313) NOISSUE - Update CI tools (absmach#2314) MG-888 - Update consumers SDK tests (absmach#2271) MG-888 - Add bootstrap SDK tests (absmach#2261) MG-2026 - Fix JSON page response key from groups to channels (absmach#2298) MG-1965 - Process Event Logs (absmach#2057) MG-1529 - User Removal (absmach#2122)

JeffMboya force-pushed the MG-2142 branch 3 times, most recently from 90b9f69 to 3198bd8 Compare April 23, 2024 11:56

JeffMboya self-assigned this Apr 23, 2024

JeffMboya force-pushed the MG-2142 branch from 16c748a to 8c72bd0 Compare April 24, 2024 12:29

arvindh123 reviewed Apr 25, 2024

View reviewed changes

bootstrap/postgres/configs.go Outdated Show resolved Hide resolved

arvindh123 reviewed Apr 25, 2024

View reviewed changes

JeffMboya marked this pull request as ready for review April 26, 2024 10:10

arvindh123 reviewed Apr 26, 2024

View reviewed changes

bootstrap/postgres/configs.go Outdated Show resolved Hide resolved

arvindh123 reviewed Apr 26, 2024

View reviewed changes

bootstrap/postgres/configs.go Outdated Show resolved Hide resolved

arvindh123 requested changes Apr 26, 2024

View reviewed changes

arvindh123 reviewed Apr 26, 2024

View reviewed changes

bootstrap/events/consumer/streams.go Outdated Show resolved Hide resolved

arvindh123 requested changes Apr 26, 2024

View reviewed changes

JeffMboya force-pushed the MG-2142 branch 2 times, most recently from 0b87686 to 9380eef Compare May 6, 2024 12:36

JeffMboya force-pushed the MG-2142 branch 3 times, most recently from 15ddd1f to 068d129 Compare May 8, 2024 11:25

JeffMboya changed the title ~~MG-2142 - Consume Things connect event in bootstrap~~ MG-2142 - Consume Things connect/disconnect event in bootstrap May 12, 2024

JeffMboya force-pushed the MG-2142 branch 6 times, most recently from b78c9aa to f6aa2fe Compare May 13, 2024 09:57

dborovcanin force-pushed the MG-2142 branch from f6aa2fe to 87860ef Compare May 15, 2024 10:57

rodneyosodo requested changes May 17, 2024

View reviewed changes

JeffMboya force-pushed the MG-2142 branch 2 times, most recently from 386c14c to 73f7017 Compare May 20, 2024 07:32

JeffMboya force-pushed the MG-2142 branch from 41c7daf to fbad36f Compare June 20, 2024 05:22

dborovcanin force-pushed the MG-2142 branch from fbad36f to 68aa366 Compare June 20, 2024 11:51

arvindh123 requested changes Jun 21, 2024

View reviewed changes

JeffMboya force-pushed the MG-2142 branch from 68aa366 to 40aabf5 Compare June 21, 2024 08:32

JeffMboya added 20 commits June 21, 2024 11:38

Refactor connectionEvent in events.go

8c1a7da

Signed-off-by: JeffMboya <[email protected]>

Refactor: revert to thingID and channelID

5ce24ed

Signed-off-by: JeffMboya <[email protected]> Refactor: revert to thingID and channelID Signed-off-by: JeffMboya <[email protected]>

Refactor: removed saved method

12f3a44

Signed-off-by: JeffMboya <[email protected]>

Refactor: Add tests

3cd7e06

Signed-off-by: JeffMboya <[email protected]>

Refactor: Add tests

8b9ea2a

Signed-off-by: JeffMboya <[email protected]>

refactor: ConnectThing and DisconnectThing methods

93ff487

Signed-off-by: JeffMboya <[email protected]>

fix: fix ConnectThing and DisconnectThing comments

3b4b78a

Signed-off-by: JeffMboya <[email protected]>

refactor: ConnectThing and DisconnectThing tests

c95639c

Signed-off-by: JeffMboya <[email protected]>

refactor: ConnectThing and DisconnectThing tests

867b4c1

Signed-off-by: JeffMboya <[email protected]>

refactor: ConnectThing and DisconnectThing tests

1f199a0

Signed-off-by: JeffMboya <[email protected]>

refactor: ConnectThing and DisconnectThing tests

75db138

Signed-off-by: JeffMboya <[email protected]>

fix: Add logging for error rollback in Save method

38f1361

Signed-off-by: JeffMboya <[email protected]>

refactor: defer rollback

dff7e71

Signed-off-by: JeffMboya <[email protected]>

refactor: Update thingID to thingIDs in connectionEvent struct

d602be3

Signed-off-by: JeffMboya <[email protected]>

refactor: use ExecContext for updating the state of configs

1eb9a61

Signed-off-by: JeffMboya <[email protected]>

refactor: update bootstrap service tests

a636ca5

Signed-off-by: JeffMboya <[email protected]>

refactor: Update ConnectThing and DisconnectThing tests

bf4aec6

Signed-off-by: JeffMboya <[email protected]>

refactor: Update ConnectThing and DisconnectThing

3a30ca5

Signed-off-by: JeffMboya <[email protected]>

JeffMboya force-pushed the MG-2142 branch from 40aabf5 to 3a30ca5 Compare June 21, 2024 08:38

arvindh123 approved these changes Jun 21, 2024

View reviewed changes

dborovcanin approved these changes Jun 24, 2024

View reviewed changes

dborovcanin merged commit 94053f3 into absmach:main Jun 24, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MG-2142 - Consume Things connect/disconnect event in bootstrap #2192

MG-2142 - Consume Things connect/disconnect event in bootstrap #2192

JeffMboya commented Apr 19, 2024 •

edited

Loading

arvindh123 left a comment

JeffMboya commented Apr 26, 2024

arvindh123 left a comment •

edited

Loading

arvindh123 left a comment •

edited

Loading

rodneyosodo May 17, 2024

arvindh123 Jun 21, 2024

arvindh123 Jun 21, 2024

JeffMboya Jun 21, 2024

arvindh123 commented Jun 21, 2024 •

edited

Loading

MG-2142 - Consume Things connect/disconnect event in bootstrap #2192

MG-2142 - Consume Things connect/disconnect event in bootstrap #2192

Conversation

JeffMboya commented Apr 19, 2024 • edited Loading

What type of PR is this?

What does this do?

Which issue(s) does this PR fix/relate to?

Have you included tests for your changes?

Did you document any new/modified feature?

arvindh123 left a comment

Choose a reason for hiding this comment

JeffMboya commented Apr 26, 2024

arvindh123 left a comment • edited Loading

Choose a reason for hiding this comment

arvindh123 left a comment • edited Loading

Choose a reason for hiding this comment

rodneyosodo May 17, 2024

Choose a reason for hiding this comment

arvindh123 Jun 21, 2024

Choose a reason for hiding this comment

arvindh123 Jun 21, 2024

Choose a reason for hiding this comment

JeffMboya Jun 21, 2024

Choose a reason for hiding this comment

arvindh123 commented Jun 21, 2024 • edited Loading

JeffMboya commented Apr 19, 2024 •

edited

Loading

arvindh123 left a comment •

edited

Loading

arvindh123 left a comment •

edited

Loading

arvindh123 commented Jun 21, 2024 •

edited

Loading