Microsoft.Azure.Cosmos.CosmosException Conflict (409) on client.close()

[BillmanH]

Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Conflict (409)

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

similar in some ways to this SO issue

Minimal steps to reproduce

This doesn't happen consistently, but in some connections it won't close the connection on command. Instead it seems like the client is trying to reference something that already exists. The odd thing is that it does exist, because it should. I'm trying to close it.

NOTE that this happens when closing a connection after all queries have run successfully and returned valid actions.

Here is a short version of the process:

from gremlin_python.driver import client

c = client.Client(
                self.endpoint,
                "g",
                username=params.username,
                password=params.password,
                message_serializer=serializer.GraphSONSerializersV2d0(),
)

callback = c.submitAsync(query)
res = c.result().all().result()

c.close()

The actual process is in upload_data which is found in the application here. The upload process works in repl, in some cases, but not others.
I'd love it if there was some way to handle this issue. Any ideas?

Any log messages given by the failure

{
    "Summary": {
        "DirectCalls": {
            "(409, 0)": 1
        },
        "RegionsContacted": 1
    },
    "name": "CreateItemStreamAsync",
    "id": "4fcdf946-a4ca-4232-b5c7-70cc55648438",
    "start time": "06:59:58:243",
    "duration in milliseconds": 21.4706,
    "data": {
        "Client Configuration": {
            "Client Created Time Utc": "2023-02-24T17:36:24.2632966Z",
            "NumberOfClientsCreated": 2458,
            "NumberOfActiveClients": 17,
            "User Agent": "cosmos-netstandard-sdk/3.26.1P|3.18.0|10|X64|Microsoft Windows 10.0.17763|.NET 6.0.13|S|F 00000010|CosmosDBGatewayService1-ComputeGateway-cdb-ms-prod-westus2-fe1",
            "ConnectionConfig": {
                "gw": "(cps:50, urto:60, p:False, httpf: True)",
                "rntbd": "(cto: 5, icto: -1, mrpc: 6, mcpe: 64, erd: True, pr: ReuseUnicastPort)",
                "other": "(ed:False, be:False)"
            },
            "ConsistencyConfig": "(consistency: NotSet, prgns:[])"
        }
    },
    "children": [
        {
            "name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
            "id": "362aa4c9-dbc9-489f-9275-1011abf809fd",
            "start time": "06:59:58:243",
            "duration in milliseconds": 21.458,
            "children": [
                {
                    "name": "Microsoft.Azure.Cosmos.Friends.DelayEstimateRequestHandler",
                    "id": "4c01522b-f312-4268-957b-67117712376c",
                    "start time": "06:59:58:243",
                    "duration in milliseconds": 21.4363,
                    "children": [
                        {
                            "name": "Microsoft.Azure.Cosmos.Handlers.RetryHandler",
                            "id": "ba8ab12b-2007-4fad-8b40-d1a13d139fc8",
                            "start time": "06:59:58:243",
                            "duration in milliseconds": 21.4334,
                            "children": [
                                {
                                    "name": "Microsoft.Azure.Cosmos.Handlers.RouterHandler",
                                    "id": "245bf730-008a-4a8b-8296-3345c9fce53e",
                                    "start time": "06:59:58:243",
                                    "duration in milliseconds": 21.4093,
                                    "children": [
                                        {
                                            "name": "Microsoft.Azure.Cosmos.Handlers.TransportHandler",
                                            "id": "9f61c3d7-0c58-4abb-ac01-4886a25e0d66",
                                            "start time": "06:59:58:243",
                                            "duration in milliseconds": 21.4081,
                                            "children": [
                                                {
                                                    "name": "Microsoft.Azure.Documents.ServerStoreModel Transport Request",
                                                    "id": "3327e8c4-f837-4a2e-9749-626c73784ccd",
                                                    "start time": "06:59:58:243",
                                                    "duration in milliseconds": 21.3851,
                                                    "data": {
                                                        "Client Side Request Stats": {
                                                            "Id": "AggregatedClientSideRequestStatistics",
                                                            "ContactedReplicas": [
                                                                {
                                                                    "Count": 1,
                                                                    "Uri": "rntbd://cdb-ms-prod-westus2-fd1.documents.azure.com:14051/apps/2768abbb-3f7b-4ab0-83e7-8de6b32c5c3b/services/3abb0032-65f9-484e-8dbf-8ed11d2d8fec/partitions/1adf7804-871e-4d4f-ad6b-3b8d046c3729/replicas/133169043154705079s/"
                                                                },
                                                                {
                                                                    "Count": 1,
                                                                    "Uri": "rntbd://cdb-ms-prod-westus2-fd1.documents.azure.com:14140/apps/2768abbb-3f7b-4ab0-83e7-8de6b32c5c3b/services/3abb0032-65f9-484e-8dbf-8ed11d2d8fec/partitions/1adf7804-871e-4d4f-ad6b-3b8d046c3729/replicas/133038479684164284s/"
                                                                },
                                                                {
                                                                    "Count": 1,
                                                                    "Uri": "rntbd://cdb-ms-prod-westus2-fd1.documents.azure.com:14403/apps/2768abbb-3f7b-4ab0-83e7-8de6b32c5c3b/services/3abb0032-65f9-484e-8dbf-8ed11d2d8fec/partitions/1adf7804-871e-4d4f-ad6b-3b8d046c3729/replicas/133197079950389549s/"
                                                                }
                                                            ],
                                                            "RegionsContacted": [
                                                                "https://graph-exodestiny-westus2.documents.azure.com/"
                                                            ],
                                                            "FailedReplicas": [],
                                                            "AddressResolutionStatistics": [],
                                                            "StoreResponseStatistics": [
                                                                {
                                                                    "ResponseTimeUTC": "2023-02-25T18:59:58.2645025Z",
                                                                    "ResourceType": "Document",
                                                                    "OperationType": "Create",
                                                                    "LocationEndpoint": "<redact>",
                                                                    "StoreResult": {
                                                                        "ActivityId": "cc544be3-2b5d-4312-9d86-1d3a62761138",
                                                                        "StatusCode": "Conflict",
                                                                        "SubStatusCode": "Unknown",
                                                                        "LSN": 181703,
                                                                        "PartitionKeyRangeId": "0",
                                                                        "GlobalCommittedLSN": 181703,
                                                                        "ItemLSN": -1,
                                                                        "UsingLocalLSN": false,
                                                                        "QuorumAckedLSN": 181703,
                                                                        "SessionToken": "-1#181703",
                                                                        "CurrentWriteQuorum": 3,
                                                                        "CurrentReplicaSetSize": 4,
                                                                        "NumberOfReadRegions": 0,
                                                                        "IsValid": true,
                                                                        "StorePhysicalAddress": "rntbd://cdb-ms-prod-westus2-fd1.documents.azure.com:14447/apps/2768abbb-3f7b-4ab0-83e7-8de6b32c5c3b/services/3abb0032-65f9-484e-8dbf-8ed11d2d8fec/partitions/1adf7804-871e-4d4f-ad6b-3b8d046c3729/replicas/133052631433602073p/",
                                                                        "RequestCharge": 4.52,
                                                                        "RetryAfterInMs": null,
                                                                        "BELatencyInMs": "0.913",
                                                                        "transportRequestTimeline": {
                                                                            "requestTimeline": [
                                                                                {
                                                                                    "event": "Created",
                                                                                    "startTimeUtc": "2023-02-25T18:59:58.2432929Z",
                                                                                    "durationInMs": 0.0038
                                                                                },
                                                                                {
                                                                                    "event": "ChannelAcquisitionStarted",
                                                                                    "startTimeUtc": "2023-02-25T18:59:58.2432967Z",
                                                                                    "durationInMs": 19.054
                                                                                },
                                                                                {
                                                                                    "event": "Pipelined",
                                                                                    "startTimeUtc": "2023-02-25T18:59:58.2623507Z",
                                                                                    "durationInMs": 0.1532
                                                                                },
                                                                                {
                                                                                    "event": "Transit Time",
                                                                                    "startTimeUtc": "2023-02-25T18:59:58.2625039Z",
                                                                                    "durationInMs": 1.9198
                                                                                },
                                                                                {
                                                                                    "event": "Received",
                                                                                    "startTimeUtc": "2023-02-25T18:59:58.2644237Z",
                                                                                    "durationInMs": 0.0699
                                                                                },
                                                                                {
                                                                                    "event": "Completed",
                                                                                    "startTimeUtc": "2023-02-25T18:59:58.2644936Z",
                                                                                    "durationInMs": 0
                                                                                }
                                                                            ],
                                                                            "serviceEndpointStats": {
                                                                                "inflightRequests": 1,
                                                                                "openConnections": 1
                                                                            },
                                                                            "connectionStats": {
                                                                                "waitforConnectionInit": "True",
                                                                                "callsPendingReceive": 0,
                                                                                "lastSendAttempt": "2023-02-25T18:59:58.2607262Z",
                                                                                "lastSend": "2023-02-25T18:59:58.2608028Z",
                                                                                "lastReceive": "2023-02-25T18:59:58.2613807Z"
                                                                            },
                                                                            "requestSizeInBytes": 879,
                                                                            "requestBodySizeInBytes": 455,
                                                                            "responseMetadataSizeInBytes": 181,
                                                                            "responseBodySizeInBytes": 65
                                                                        },
                                                                        "TransportException": null
                                                                    }
                                                                }
                                                            ]
                                                        }
                                                    }
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}

Expected/desired behavior

Connection closes. No crash.

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
10

Versions

tried on both:

gremlinpython==3.5.1
gremlinpython==3.6.0

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

[BillmanH]

I think I may have figured it out. Redesigning and removing many edges seems to have prevented this from happening. However it's difficult to verify.

Is there any way that I can take that id 4fcdf946-a4ca-4232-b5c7-70cc55648438 and look it up somewhere? If there was some process where I can look up the ID and get the offending query that would be really awesome. Is there any way to do this in Azure Cosmos?

This would be especially helpful for large scale (cosmsose sized) operations where it will be difficult to track individual queries.