Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sending Single Message Per Client Prefix/ User Prefix #107

Merged
merged 4 commits into from
Jun 18, 2018

Conversation

gekco
Copy link
Contributor

@gekco gekco commented Jun 5, 2018

previously for each channel subscribed to a group message was being pushed to list stored at channel key. I have grouped those messages by sending list of channels associated with that that channel redis key with a single message . It saves a great deal of time by reducing the number of push and pops from redis.
Fixes : #7, #83

Copy link
Member

@andrewgodwin andrewgodwin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not entirely clear what this is doing - the code is quite hard to read. Could you add some comments and make it very clear what's happening?

redis.call('EXPIRE', KEYS[i], %d)
end
end
""" % self.expiry
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you indent this without changing it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i guess the IDE did it. i will revert it.

channel_key_to_message[key]["__asgi_channel__"] = ",".join(channel_key_to_message[key]["__asgi_channel__"])
channel_key_to_message[key] = self.serialize(channel_key_to_message[key])

return channel_keys, connection_to_channel_keys, channel_key_to_message, channel_key_to_capacity
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block needs some serious comments to say what's happening - I'm finding it hard to read.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on it

@gekco
Copy link
Contributor Author

gekco commented Jun 10, 2018

(Correct me if i am understanding this wrong)
say there is a group name 'Group-A' with 100 members(channels)

originally when i called group_send('Group-A', message_dictionary)
the same message was being pushed 100 times each time with a different asgi_channel value to the same redis queue(because multiple channels MAY have same redis key) which was being popped at function receive_single 100 times obviously and the message was getting processed for a single channel.

Modifications:
what i did was coalesce those messages i.e. for those 100 channels i will be sending a single message per unique redis key K for all the channels that are mapped to K, by sending a comma separated string of those channels instead of a single channel in asgi_channel.

On the receiving end i just process the message for all of the channels that are received in the message in a for loop. This Largely reduces the time to push group messages onto channel redis layer and the number of pops.
For Example if all the 100 channels have the same channel key there will be a single message for all the 100 channels which will be pushed on the redis queue at channel key and popped from the same.


# channel_keys does not contain a single redis key more than once
async with self.connection(connection_index) as connection:
await connection.eval(group_send_lua, keys=channel_redis_keys, args=args)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

previously channel_redis_keys would contain duplicate keys as multiple channels may map to same redis key. hence the more number of pushes.

@andrewgodwin
Copy link
Member

OK, I mostly understand it now, but a couple of things I'd like to see changed:

  • The _map_channel_to_connection function is now far too complex - that whole thing should be refactored so it doesn't have to pass around things as dicts of lists and so forth. Maybe flatten it more into the only place it's called.

  • I'm not particularly happy with the comma-separated approach to the storing of channel names. Maybe just use a native msgpack list?

@gekco
Copy link
Contributor Author

gekco commented Jun 12, 2018

  1. I have restructured my code a bit , also using a more relevant function name, eliminated some repitive code, flattened it, have a look. i am now using _map_channel_keys_to_connection function.
  2. I am using a msgpack native list now

@andrewgodwin andrewgodwin merged commit 9f05013 into django:master Jun 18, 2018
@andrewgodwin
Copy link
Member

Thanks!

@gekco
Copy link
Contributor Author

gekco commented Jun 18, 2018

@andrewgodwin Thank you so much. This was my first PR to get merged . Yayy. :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants