django.core.exceptions.SynchronousOnlyOperation after calling .group_send

[MarlonCorreia]

OS

• MacOs Monterey
• Amazon Linux AMI 2018.03

PIP Freeze

List

• aioredis==1.3.1
• amqp==5.1.1
• appnope==0.1.3
• asgiref==3.5.2
• async-timeout==4.0.2
• attrs==21.4.0
• autobahn==22.6.1
• Automat==20.2.0
• aws-xray-sdk==2.4.2
• backcall==0.2.0
• billiard==3.6.4.0
• boto3==1.24.37
• botocore==1.27.37
• CacheControl==0.12.11
• cachetools==4.2.4
• celery==5.2.3
• certifi==2022.6.15
• cffi==1.15.1
• channels==3.0.4
• channels-redis==3.3.0
• charset-normalizer==2.0.12
• click==8.1.3
• click-didyoumean==0.3.0
• click-plugins==1.1.1
• click-repl==0.2.0
• coconutpy==2.4.3
• constantly==15.1.0
• cryptography==37.0.4
• daphne==3.0.2
• decorator==5.1.1
• defusedxml==0.7.1
• dj-database-url==0.5.0
• Django==3.2.6
• django-allauth==0.51.0
• django-amazon-ses==4.0.0
• django-colorfield==0.2.1
• django-cors-headers==3.7.0
• django-db-geventpool==4.0.0
• django-dotenv==1.4.2
• django-filter==2.4.0
• django-nested-admin==3.3.3
• django-redis==5.0.0
• django-rest-auth==0.9.5
• django-storages==1.11.1
• django-tenants==3.3.2
• djangorestframework==3.12.4
• djangorestframework-gis==0.17
• drf-nested-routers==0.93.3
• firebase-admin==4.0.0
• future==0.18.2
• gevent==20.12.1
• google-api-core==1.32.0
• google-api-python-client==2.54.0
• google-auth==1.35.0
• google-auth-httplib2==0.1.0
• google-cloud-core==2.3.2
• google-cloud-firestore==2.6.0
• google-cloud-storage==2.4.0
• google-crc32c==1.3.0
• google-resumable-media==2.3.3
• googleapis-common-protos==1.56.4
• googlemaps==4.6.0
• greenlet==1.1.2
• grpcio==1.47.0
• gunicorn==20.0.4
• h3==3.7.4
• hiredis==2.0.0
• httplib2==0.20.4
• hyperlink==21.0.0
• idna==3.3
• importlib-metadata==4.13.0
• incremental==21.3.0
• ipython==7.7.0
• ipython-genutils==0.2.0
• jedi==0.18.1
• jmespath==1.0.1
• jsonpickle==2.2.0
• kombu==5.2.4
• msgpack==1.0.4
• newrelic==6.8.0.163
• numpy==1.23.1
• oauthlib==3.2.0
• packaging==21.3
• pandas==1.4.4
• parso==0.8.3
• pexpect==4.8.0
• pickleshare==0.7.5
• Pillow==8.2.0
• prompt-toolkit==2.0.10
• proto-plus==1.20.6
• protobuf==3.20.1
• psycogreen==1.0.2
• psycopg2-binary==2.9.3
• ptyprocess==0.7.0
• pyasn1==0.4.8
• pyasn1-modules==0.2.8
• pycparser==2.21
• Pygments==2.12.0
• PyJWT==2.4.0
• pyOpenSSL==22.0.0
• pyparsing==3.0.9
• python-dateutil==2.8.2
• python-monkey-business==1.0.0
• python3-openid==3.2.0
• pytz==2022.1
• redis==3.5.3
• requests==2.26.0
• requests-oauthlib==1.3.1
• rsa==4.9
• s3-environ==0.0.5
• s3transfer==0.6.0
• sentry-sdk==1.3.1
• service-identity==21.1.0
• Shapely==1.8.2
• six==1.16.0
• sqlparse==0.4.2
• timezonefinder==6.0.1
• traitlets==5.3.0
• Twisted==22.4.0
• txaio==22.2.1
• typing_extensions==4.3.0
• tzwhere==3.0.3
• uritemplate==4.1.1
• urllib3==1.26.11
• uuid==1.30
• vine==5.0.0
• wcwidth==0.2.5
• whitenoise==5.0.1
• wrapt==1.14.1
• zipp==3.8.1
• zope.event==4.5.0
• zope.interface==5.4.0

Problem

While testing, it seems that when we have some numbers of requests at the same time, the exception SynchronousOnlyOperation is thrown, after testing for a bit, I found that this part of the code here:

def notify_websocket_of_update(self, id, updated):
    channel_layer = get_channel_layer()
    payload = self.build_websocket_payload(id, 'update', updated)
    async_to_sync(channel_layer.group_send)("name", payload)

and when I comment the async_to_sync(channel_layer.group_send)("name", payload) line, the problem stops from happening. Which makes me almost certain that the problems happens because of the async context inside .group_send

To me, it seems that when we hit some async event_loop other requests coming in that are reusing the thread stumbles to the exception. But then I thought that the async_to_sync should solve this problem, or am I wrong?

At this point, I'm mainly checking if this is a common behavior, or if anyone had face this problem before and could share a fix.

StackTrace

I know this StackTrace is not directly related to the code snippet I put above, but it's an example of the problem that is created when I have the channel group_send method call (when I remove the .group_send call, this exception never happens).

Traceback (most recent call last):
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django/utils/deprecation.py", line 116, in __call__
    response = self.process_request(request)
  File "/Users/ms/team/project/helpers/tenant.py", line 33, in process_request
    tenant = self.get_tenant(domain_model, hostname)
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django_tenants/middleware/main.py", line 27, in get_tenant
    domain = domain_model.objects.select_related('tenant').get(domain=hostname)
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
    cursor = self.connection.cursor()
  File "/Users/ms/team/project/venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 24, in inner
    raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

Service

The service is started using gunicorn, on a wsgi server.