Django Cursor not found on publish

[Bertus-W]

Version
Please provide the versions of the pulpcore and pulp_rpm packages in use, and how they are installed. If you are using Pulp via Katello, please provide the Katello version.

"versions": [
    {
      "component": "core",
      "version": "3.18.6"
    },
    {
      "component": "rpm",
      "version": "3.17.10"
    },
    {
      "component": "deb",
      "version": "2.18.0"
    }
  ],

Describe the bug

"error": {
    "traceback": "  File \"/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 410, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py\", line 389, in publish\n    generate_repo_metadata(\n  File \"/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py\", line 550, in generate_repo_metadata\n    for package in packages.order_by(\"name\", \"evr\").iterator():\n  File \"/usr/lib/python3.9/site-packages/django/db/models/query.py\", line 353, in _iterator\n    yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)\n  File \"/usr/lib/python3.9/site-packages/django/db/models/query.py\", line 51, in __iter__\n    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)\n  File \"/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1178, in execute_sql\n    cursor.close()\n",
    "description": "cursor \"_django_curs_140546373439744_sync_5\" does not exist\n"
  }

To Reproduce
Steps to reproduce the behavior:
Publish a very large repository with remote links

Expected behavior
A clear and concise description of what you expected to happen.
To publish without issue

Additional context
Add any other context about the problem here. Please provide links to any previous discussions via Discourse or Bugzilla.
Can someone tell me how I can fix this without recreating the repository?

[dralley]

very large repository

Can you elaborate? Is this a particular repo, a custom one, how large is it?

[Bertus-W]

hi, sorry, the repository got 10.000 packages and basically contains the alma8 baseos and alma8 appstream remote repositories which get synced regulary. Do you know how to fix the current condition I'm having? repair api didn't do anything unfortunately

[Bertus-W]

I tried restarting postgres, all my pulp3 services. but that didn't help. restarting my whole pulp server DID fix it for now.

[dralley]

Are you using pgBouncer?

[Bertus-W]

No I’m not. I don’t have any modifications beyond the basic setup

[dralley]

@ipanova I'm maybe 70% confident this is related to the issue you've just fixed. WDYT?

[ipanova]

My changes did not really touch that part, however what strikes me odd is that pulp services + postgres restart did not help until whole server needed to be rebooted. If the user did not experience this issue after server reboot maybe this weird behavior should be chalked up to some server malfunction state at the moment.

[Bertus-W]

Restarting postgres does seem to have solved the issue a while back. Have only experienced it once more where I could test that maybe I didn’t restart postgres properly the first time. It’s a weird issue

[dralley]

How much memory does the system have? Maybe it's possible that the OOM killer came along and killed the posgresql process?

[Bertus-W]

8GB total. Currently 2.8GB free. Don’t know about then. It’s a VM with the sole purpose of running pulp so it isn’t used for something else at any point. Postgres version is 10.21-2 if that helps

[dralley]

That's not much free memory, particularly if you want to run more than one task at a time. Maybe you can tweak the worker / webserver counts to free up a bit more space for tasks?

[dralley]

Postgres version is 10.21-2 if that helps

I know we have issues on Postgresql version 10 that we have never seen on Postgresql version 12+, we recommend version 12+ in the docs

I don't know if this has ever recurred, but I do recommend a newer Postgresql if possible.

[ggainey]

We're going to close this issue - esp since we have never seen it on postgres-12, which is the current min recommendation. If you hit a more-reliable reproducer, please do reopen!