Cannot return null for non-nullable field Token.asset.

[zxpectre]

Summary

Same query we were using since ages is now triggering some errors along with full or partial response data.

This on all our mainnet instances, some running latest versions 8.3.2, some with previous 8.3.1

Steps to reproduce the bug

fragment _GetTransactions on Query {
  transactions(limit: $limit, order_by: $order_by, offset: $offset, where: $where) {
    blockIndex
    fee
    hash
    invalidBefore
    invalidHereafter
    includedAt
    size
    totalOutput
    deposit
    metadata {
      key
      value
      __typename
    }
    inputs {
      address
      value
      txHash
      tokens {
        asset {
          assetId
          policyId
          assetName
          fingerprint
          __typename
        }
        quantity
        __typename
      }
      sourceTxHash
      sourceTxIndex
      redeemer {
        fee
        index
        purpose
        scriptHash
        unitMem
        unitSteps
        __typename
      }
      __typename
    }
    outputs {
      address
      value
      txHash
      tokens {
        asset {
          assetId
          policyId
          assetName
          fingerprint
          __typename
        }
        quantity
        __typename
      }
      script {
        hash
        serialisedSize
        type
        __typename
      }
      __typename
    }
    mint {
      asset {
        assetId
        policyId
        assetName
        fingerprint
        __typename
      }
      quantity
      __typename
    }
    scriptSize
    validContract
    referenceInputs {
      address
      sourceTxHash
      sourceTxIndex
      tokens {
        asset {
          assetId
          policyId
          assetName
          fingerprint
          __typename
        }
        quantity
        __typename
      }
      txHash
      value
      __typename
    }
    collateralInputs {
      address
      sourceTxHash
      sourceTxIndex
      tokens {
        asset {
          assetId
          policyId
          assetName
          fingerprint
          __typename
        }
        quantity
        __typename
      }
      txHash
      value
      __typename
    }
    collateralOutputs {
      address
      addressHasScript
      datum {
        bytes
        hash
        value
        __typename
      }
      index
      paymentCredential
      script {
        hash
        serialisedSize
        type
        __typename
      }
      txHash
      value
      __typename
    }
    scripts {
      hash
      serialisedSize
      type
      __typename
    }
    redeemers {
      fee
      index
      purpose
      scriptHash
      unitMem
      unitSteps
      __typename
    }
    __typename
  }
  __typename
}

fragment _GetTip on Query {
  cardano {
    tip {
      number
      slotNo
      slotInEpoch
      epochNo
      forgedAt
      __typename
    }
    __typename
  }
  __typename
}

query GetTransactions($limit: Int, $order_by: [Transaction_order_by!], $offset: Int, $where: Transaction_bool_exp) {
  ..._GetTransactions
  ..._GetTip
}

{
  "where": {
    "hash": {
      "_in": [
        "13800d6782e4a8a63ab86c17fe19082960daa544992adcb8e31f4093ba749a3e",
        "bbb69ff9ebc2c61fada3f91ad1d1c864f29a94c4ec8d065db6346e3b67d1fa85",
        "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd"
      ]
    }
  }
}

Actual Result

{
  "errors": [
    {
      "message": "Cannot return null for non-nullable field Token.asset.",
      "locations": [
        {
          "line": 50,
          "column": 9
        }
      ],
      "path": [
        "transactions",
        1,
        "outputs",
        0,
        "tokens",
        0,
        "asset"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR",
        "exception": {
          "stacktrace": [
            "Error: Cannot return null for non-nullable field Token.asset.",
            "    at completeValue (/app/node_modules/graphql/execution/execute.js:560:13)",
            "    at completeValueCatchingError (/app/node_modules/graphql/execution/execute.js:495:19)",
            "    at resolveField (/app/node_modules/graphql/execution/execute.js:435:10)",
            "    at executeFields (/app/node_modules/graphql/execution/execute.js:275:18)",
            "    at collectAndExecuteSubfields (/app/node_modules/graphql/execution/execute.js:713:10)",
            "    at completeObjectValue (/app/node_modules/graphql/execution/execute.js:703:10)",
            "    at completeValue (/app/node_modules/graphql/execution/execute.js:591:12)",
            "    at completeValue (/app/node_modules/graphql/execution/execute.js:557:21)",
            "    at completeValueCatchingError (/app/node_modules/graphql/execution/execute.js:495:19)",
            "    at /app/node_modules/graphql/execution/execute.js:618:25"
          ]
        }
      }
    },
    {
      "message": "Cannot return null for non-nullable field Token.asset.",
      "locations": [
        {
          "line": 69,
          "column": 7
        }
      ],
      "path": [
        "transactions",
        1,
        "mint",
        0,
        "asset"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR",
        "exception": {
          "stacktrace": [
            "Error: Cannot return null for non-nullable field Token.asset.",
            "    at completeValue (/app/node_modules/graphql/execution/execute.js:560:13)",
            "    at completeValueCatchingError (/app/node_modules/graphql/execution/execute.js:495:19)",
            "    at resolveField (/app/node_modules/graphql/execution/execute.js:435:10)",
            "    at executeFields (/app/node_modules/graphql/execution/execute.js:275:18)",
            "    at collectAndExecuteSubfields (/app/node_modules/graphql/execution/execute.js:713:10)",
            "    at completeObjectValue (/app/node_modules/graphql/execution/execute.js:703:10)",
            "    at completeValue (/app/node_modules/graphql/execution/execute.js:591:12)",
            "    at completeValue (/app/node_modules/graphql/execution/execute.js:557:21)",
            "    at completeValueCatchingError (/app/node_modules/graphql/execution/execute.js:495:19)",
            "    at /app/node_modules/graphql/execution/execute.js:618:25"
          ]
        }
      }
    }
  ],
  "data": {
    "transactions": [
      {
        "blockIndex": 24,
        "fee": "180593",
        "hash": "13800d6782e4a8a63ab86c17fe19082960daa544992adcb8e31f4093ba749a3e",
        "invalidBefore": null,
        "invalidHereafter": null,
        "includedAt": "2024-12-20T23:08:42Z",
        "size": 570,
        "totalOutput": "16819407",
        "deposit": "0",
        "metadata": [
          {
            "key": "73",
            "value": {
              "v": "1.0",
              "tags": [
                "smart-send"
              ],
              "type": "tx",
              "title": "Test",
              "referrer": ""
            },
            "__typename": "TransactionMetadata"
          }
        ],
        "inputs": [
          {
            "address": "addr1q9z3wmu6qcaxcj6f2gl4d8fesc89s7dujs72z2sdqdrs9jkagkyc6hn3tm554wrncavlnjsedvcf7d8h5wsp658r9lts0rt4qq",
            "value": "17000000",
            "txHash": "13800d6782e4a8a63ab86c17fe19082960daa544992adcb8e31f4093ba749a3e",
            "tokens": [],
            "sourceTxHash": "8a23876e13b4f7161d7b12afa038f7a17ac4982a691bdd6272b9d5213cbd6c0e",
            "sourceTxIndex": 5,
            "redeemer": null,
            "__typename": "TransactionInput"
          }
        ],
        "outputs": [
          {
            "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
            "value": "1000000",
            "txHash": "13800d6782e4a8a63ab86c17fe19082960daa544992adcb8e31f4093ba749a3e",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9z3wmu6qcaxcj6f2gl4d8fesc89s7dujs72z2sdqdrs9jkagkyc6hn3tm554wrncavlnjsedvcf7d8h5wsp658r9lts0rt4qq",
            "value": "2000000",
            "txHash": "13800d6782e4a8a63ab86c17fe19082960daa544992adcb8e31f4093ba749a3e",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9z3wmu6qcaxcj6f2gl4d8fesc89s7dujs72z2sdqdrs9jkagkyc6hn3tm554wrncavlnjsedvcf7d8h5wsp658r9lts0rt4qq",
            "value": "3000000",
            "txHash": "13800d6782e4a8a63ab86c17fe19082960daa544992adcb8e31f4093ba749a3e",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9z3wmu6qcaxcj6f2gl4d8fesc89s7dujs72z2sdqdrs9jkagkyc6hn3tm554wrncavlnjsedvcf7d8h5wsp658r9lts0rt4qq",
            "value": "5000000",
            "txHash": "13800d6782e4a8a63ab86c17fe19082960daa544992adcb8e31f4093ba749a3e",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9z3wmu6qcaxcj6f2gl4d8fesc89s7dujs72z2sdqdrs9jkagkyc6hn3tm554wrncavlnjsedvcf7d8h5wsp658r9lts0rt4qq",
            "value": "5819407",
            "txHash": "13800d6782e4a8a63ab86c17fe19082960daa544992adcb8e31f4093ba749a3e",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          }
        ],
        "mint": [],
        "scriptSize": 0,
        "validContract": true,
        "referenceInputs": [],
        "collateralInputs": [],
        "collateralOutputs": [],
        "scripts": [],
        "redeemers": [],
        "__typename": "Transaction"
      },
      null,
      {
        "blockIndex": 33,
        "fee": "291781",
        "hash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
        "invalidBefore": "139943990",
        "invalidHereafter": "140030390",
        "includedAt": "2024-11-13T15:06:46Z",
        "size": 3099,
        "totalOutput": "20475491",
        "deposit": "0",
        "metadata": [
          {
            "key": "724055454845654",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "725320000000000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "726201255575092",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "726324852812301",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "729003662923607",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "732843000708823",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "733800000000000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "734500028869859",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "737394576365298",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "738135389361642",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "739127689279275",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "744818430000000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "747311701117701",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "750030991961946",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "751361047081495",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "751431619175030",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "752351336654013",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "752780526864227",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "753058773154100",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "754916272197078",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "754919561704000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "757335790000000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "759398067380222",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "760622434301000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "761133326530000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "762700000000000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "763196200986605",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "763523336683119",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "764768836534873",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "765251300000000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "767824284143732",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "769951972600000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "721313098431224",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "73",
            "value": {
              "v": "1.0",
              "tags": [
                "gcfs",
                "2abaf6d73e5838cff911cdf8f6629cff1e1955df3afb89738e248f8e",
                "examples",
                "asset1kc44y97vj7rpaswsdtv30e3t5ascq9jjvqjydw",
                "header"
              ],
              "type": "tx",
              "group": [
                "2abaf6d73e5838cff911cdf8f6629cff1e1955df3afb89738e248f8e-example",
                "s-2024-11-13T15:04:50.394Z"
              ],
              "index": 1,
              "title": "GCFS #2: Header ",
              "parent": "a99e17bf7266f9d266ce0d4dc52f74e13f124cce32a250e1053e3309fb33469b"
            },
            "__typename": "TransactionMetadata"
          },
          {
            "key": "4237",
            "value": {
              "2abaf6d73e5838cff911cdf8f6629cff1e1955df3afb89738e248f8e": {
                "examples": {
                  "fs": {
                    "d": [
                      {
                        "d": [
                          {
                            "e": {
                              "a": [
                                [
                                  "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5",
                                  "ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr"
                                ]
                              ],
                              "h": [
                                "6ee7d77df32c1c0440a82c4d8201d2cbc509274d7f88dd302b6f06f4fb7a307d",
                                "328edf6322de1f6d1ac4cc3a1e7754a2de9282b76cab251f025c45bf42c68a42"
                              ]
                            },
                            "n": "PaymentRequest.gcscript"
                          },
                          {
                            "f": [
                              "cca8feb247b1304e1b5b295aa6730598c847340f1c2285ea142ca9c7cdf3561f",
                              "1a276935d69d47ddc0e00c8a942b5268ba0ab538035471f98069f8ab4d4685dd"
                            ],
                            "n": "twd.gcscript"
                          }
                        ],
                        "n": "src"
                      },
                      {
                        "d": [
                          {
                            "e": {
                              "a": [
                                [
                                  "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5",
                                  "ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr"
                                ]
                              ],
                              "h": [
                                "ebe3799ff427b22fddf6f4c77397732297273dca2d0ae4de3a3301e0204048ec",
                                "04fb4c9cf096dafcfcf8a9eee430c6e3d396152c765ff3fe96a292193ccde5d5"
                              ]
                            },
                            "n": "generate10GiftWallets.html"
                          }
                        ],
                        "n": "www"
                      }
                    ],
                    "n": "/"
                  },
                  "refChk": [
                    "7b2274797065223a22426c6f6f6d46696c746572222c225f73697a65223a3239",
                    "2c225f6e62486173686573223a372c225f66696c746572223a7b2273697a6522",
                    "3a33322c22636f6e74656e74223a223632725a42513d3d227d2c225f73656564",
                    "223a3230363232333733333339323339332c225f5f76223a317d"
                  ],
                  "description": "Updated with GameChanger Wallet at Wed Nov 13 2024 11:55:37 G..."
                }
              }
            },
            "__typename": "TransactionMetadata"
          },
          {
            "key": "423704323370000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "701399439900000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "710922001398702",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711071892109067",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711179768604280",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711216408720006",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711235448711355",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711310577495875",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711517962093000",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711518539466679",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711631176158809",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "711962915950923",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "713075302670987",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "713649591624284",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "713704592450973",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "713712315501890",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "714072991667006",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "714342253432425",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "714717148666530",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "714873969322264",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "715325832026124",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "715351304780370",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "715808599261022",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "715893100205608",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "716797665776545",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "717096214904893",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "717139047758511",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "717244924736962",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "717441504849855",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "718117915150964",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "718240667081266",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "718857063358711",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "719008949073647",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "719079295898431",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "719100167209772",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "719131930517167",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "719179335982313",
            "value": 0,
            "__typename": "TransactionMetadata"
          },
          {
            "key": "719602082858423",
            "value": 0,
            "__typename": "TransactionMetadata"
          }
        ],
        "inputs": [
          {
            "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
            "value": "20767272",
            "txHash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
            "tokens": [],
            "sourceTxHash": "975fc8eb153d67f7f0ba89e4ed00d2b12be8c32ea430e6b95a99efa0a191a4fe",
            "sourceTxIndex": 30,
            "redeemer": null,
            "__typename": "TransactionInput"
          }
        ],
        "outputs": [
          {
            "address": "addr1q90wtcxyr89ps9afj9zunkks2ag2ut737a86yqm826wt9glepjs00hysz9zg9qlpyv9h90eudca80h7sy4hxyzvuvt8sz0dqpv",
            "value": "2000000",
            "txHash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
            "value": "2155080",
            "txHash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
            "tokens": [
              {
                "asset": {
                  "assetId": "2abaf6d73e5838cff911cdf8f6629cff1e1955df3afb89738e248f8e6578616d706c6573",
                  "policyId": "2abaf6d73e5838cff911cdf8f6629cff1e1955df3afb89738e248f8e",
                  "assetName": "6578616d706c6573",
                  "fingerprint": "asset1kc44y97vj7rpaswsdtv30e3t5ascq9jjvqjydw",
                  "__typename": "Asset"
                },
                "quantity": "1",
                "__typename": "Token"
              }
            ],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
            "value": "2000000",
            "txHash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
            "value": "3000000",
            "txHash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
            "value": "5000000",
            "txHash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          },
          {
            "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
            "value": "6320411",
            "txHash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
            "tokens": [],
            "script": null,
            "__typename": "TransactionOutput"
          }
        ],
        "mint": [
          {
            "asset": {
              "assetId": "2abaf6d73e5838cff911cdf8f6629cff1e1955df3afb89738e248f8e6578616d706c6573",
              "policyId": "2abaf6d73e5838cff911cdf8f6629cff1e1955df3afb89738e248f8e",
              "assetName": "6578616d706c6573",
              "fingerprint": "asset1kc44y97vj7rpaswsdtv30e3t5ascq9jjvqjydw",
              "__typename": "Asset"
            },
            "quantity": "1",
            "__typename": "Token"
          }
        ],
        "scriptSize": 0,
        "validContract": true,
        "referenceInputs": [
          {
            "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
            "sourceTxHash": "a99e17bf7266f9d266ce0d4dc52f74e13f124cce32a250e1053e3309fb33469b",
            "sourceTxIndex": 0,
            "tokens": [],
            "txHash": "dbc561fd029d0c4f8a7692fdfe175ed4bc973ab57bab1f53be00e6fa66c17ebd",
            "value": "19882030",
            "__typename": "TransactionInput"
          }
        ],
        "collateralInputs": [],
        "collateralOutputs": [],
        "scripts": [],
        "redeemers": [],
        "__typename": "Transaction"
      }
    ],
    "__typename": "Query",
    "cardano": {
      "tip": {
        "number": 11292137,
        "slotNo": "144111101",
        "slotInEpoch": "82301",
        "epochNo": 531,
        "forgedAt": "2024-12-31T20:36:32Z",
        "__typename": "Block"
      },
      "__typename": "Cardano"
    }
  }
}

Expected Result

Response without errors

Environment

https://ar02.gamechanger.finance:2087/manifest/

Platform

• Linux (Ubuntu)
• Linux (Other)
• macOS
• Windows

Platform version

Ubuntu server 20.24

Runtime

• Node.js
• Docker

Runtime version

podman version 4.9.3 | Docker Compose version v2.28.1

[zxpectre]

Sorry for the bug report on New Years Eve @Kammerlo !
I was hoping to update to latest versions to fix it but did not work and did not want to wait longer to report as these errors are shown to users

BTW Happy New Year to you, our "b*tt saviour" as I said to Nicolas Cerny on Buenos Aires

[Kammerlo]

Hey @zxpectre, Happy new Year as well :D I'm happy to help! <3
And sorry for the late reply! I will look into this issue today.
Could you share the log of the background service + server?

This issue looks like there went something wrong with the token syncing. I adjusted the way it is done last month to speed up the process.

The tokens were they newly created and queried directly after or was this an old token?

[zxpectre]

Hi @Kammerlo, any news? is not even self-healing after time passes, more queries are failing, I bet is stucked on a loop at 99% asset processing.

I do not know for certain that only new assets are the issue but it seems new mints are triggering it.

cardano-graphql-server-1  | {"name":"cardano-graphql","hostname":"50582086d7a2","pid":2,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 99%","time":"2025-01-07T13:18:00.078Z","v":0}
cardano-graphql-server-1  | {"name":"cardano-graphql","hostname":"50582086d7a2","pid":2,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2025-01-07T13:18:00.080Z","v":0}
cardano-graphql-server-1  | {"name":"cardano-graphql","hostname":"50582086d7a2","pid":2,"level":30,"module":"CardanoNodeClient","tip":{"slot":144689583,"id":"91302fcbb7d6e5480901d09acab729d1c3a18fce0241f8f7bd7de3f43e20f0c5"},"msg":"[Object: null prototype] {\n  slot: 144689583,\n  id: '91302fcbb7d6e5480901d09acab729d1c3a18fce0241f8f7bd7de3f43e20f0c5'\n}","time":"2025-01-07T13:18:00.088Z","v":0}

cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d86fd57116ae6017dd221b3ea9283b67434c3d51a4b76a32c3b7335800d9d5429e2","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.246Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"a4d9b265240df1ac3eff20cf1b532ebec28a1dbc8dba583494726847447261676f6e","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.260Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d86e20a1101764175a420cc41148eb0aeefb81a9bcfd33befb1086654db8e080923","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.278Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"09262ac4df87e3f6bc505ab5c8ff6302fc27c19456f4ba17842733464964696f74546f6b656e","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.291Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d86828c7be9b6df5eca6c6cdee7d934c9969b0ae1edf458c726b990dbbf15a5c069","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.308Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"6786d3599f071e49798c6ae7b8bc4ac1184257b27243d2d822a98e46445241474f4e","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.322Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"f5808c2c990d86da54bfc97d89cee6efa20cd8461616359478d96b4c82e2b1fd27a7712a1a9cf750dfbea1a5778611b20e06dd6a611df7a643f8cb75","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.342Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"8dcaec31847fe00ee9a9b8f854c7146b595f11bba0b6ccb3a1b360bc524f434b59","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.355Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"05a39a2c2a96934fc7c74aaa3222e59b3fe8cc0e81534ebd1a0053d6445241474f4e","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.368Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d86fce1d2106670ff894af4d4485dd37b5773a0c7644699e62ae7dedc5c0d19b74b","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.387Z","v":0}
cardano-graphql-background-1  | {"name":"background","hostname":"5aacab6ad18f","pid":2,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d862038c304ec033ec332553137b3741a17126d6a3869779019c4af67eb6b318c1c","msg":"Adding metadata to asset","time":"2025-01-07T13:19:17.401Z","v":0}

[zxpectre]

Restarted one instance, but was no good.
Other queries are failing even worse, complete broken data responses like this paymentAddresses:

{
    "errors": [
        {
            "message": "TypeError: Cannot read properties of null (reading 'assetId')",
            "locations": [
                {
                    "line": 4,
                    "column": 5
                }
            ],
            "path": [
                "paymentAddresses",
                0,
                "summary"
            ],
            "extensions": {
                "code": "INTERNAL_SERVER_ERROR",
                "exception": {
                    "stacktrace": [
                        "Error: TypeError: Cannot read properties of null (reading 'assetId')",
                        "    at /app/packages/api-cardano-db-hasura/dist/executableSchema.js:93:39",
                        "    at Generator.throw (<anonymous>)",
                        "    at rejected (/app/packages/api-cardano-db-hasura/dist/executableSchema.js:6:65)",
                        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
                    ]
                }
            }
        }
    ],
    "data": {
        "paymentAddresses": [
            {
                "address": "addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr",
                "summary": null,
                "__typename": "PaymentAddress"
            }
        ],
        "__typename": "Query",
        "cardano": {
            "tip": {
                "number": 11320877,
                "slotNo": "144691018",
                "slotInEpoch": "230218",
                "epochNo": 532,
                "forgedAt": "2025-01-07T13:41:49Z",
                "__typename": "Block"
            },
            "__typename": "Cardano"
        }
    }
}

Weird thing is that was working fine since your last update, during last weeks of 2024, not sure what changed on-chain that could cause this, or maybe is time/epoch related?. Also is a mainnet-only issue as far as we can tell.

[Kammerlo]

Hi @zxpectre
I'm still on in investigating. I started yesterday a complete new instance to check where the issue could be. Could you send me the curl for the last request?
Which services did you tried to restart?

[zxpectre]

Sure @Kammerlo, thank you! , is

curl 'https://ar01.gamechanger.finance:2053/cardano-graphql/' \
  -H 'Accept-Language: en-US,en;q=0.9,es;q=0.8' \
  -H 'Connection: keep-alive' \
  -H 'Origin: https://beta-wallet.gamechanger.finance' \
  -H 'Referer: https://beta-wallet.gamechanger.finance/' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: cross-site' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36' \
  -H 'accept: */*' \
  -H 'content-type: application/json' \
  -H 'sec-ch-ua: "Chromium";v="130", "Google Chrome";v="130", "Not?A_Brand";v="99"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Linux"' \
  --data-raw $'{"operationName":"PaymentAddresses","variables":{"addresses":["addr1q9faamq9k6557gve35amtdqph99h9q2txhz07chaxg6uwwgd6j6v0fc04n5ehg292yxvs292vesrqqmxqfnp7yuwn7yqczuqwr"]},"query":"fragment _PaymentAddresses on Query {\\n  paymentAddresses(addresses: $addresses) {\\n    address\\n    summary {\\n      assetBalances {\\n        asset {\\n          assetId\\n          policyId\\n          assetName\\n          fingerprint\\n          __typename\\n        }\\n        quantity\\n        __typename\\n      }\\n      __typename\\n    }\\n    __typename\\n  }\\n  __typename\\n}\\n\\nfragment _GetTip on Query {\\n  cardano {\\n    tip {\\n      number\\n      slotNo\\n      slotInEpoch\\n      epochNo\\n      forgedAt\\n      __typename\\n    }\\n    __typename\\n  }\\n  __typename\\n}\\n\\nquery PaymentAddresses($addresses: [String]\u0021) {\\n  ..._PaymentAddresses\\n  ..._GetTip\\n}\\n"}'

I restarted first server, background and hasura, and then the full setup with node and db-sync included. Same results not only on the restarted instance but also on all the other deployed production instances as mentioned you before. Is not a local state issue.

At this point we are not sure if we should downgrade versions now as an emergency measure or if that would not be compatible with newer node, ogmios and dbsync versions

[Kammerlo]

@zxpectre There is an issue with the postgres database. So at least in my case. Could you please check if you are getting the same logs in the postgres container
2025-01-07 17:23:25.546 UTC [534335] ERROR: could not access status of transaction 184617645 2025-01-07 17:23:25.546 UTC [534335] DETAIL: Could not open file "pg_xact/00B0": No such file or directory.
This error is hindering the background service to start.
I tried to restart the database and then shortly after the background service and the background service is currently starting, I'll keep you posted if it worked out as a work around to buy me some time to find the root cause.

[Kammerlo]

It's still stuck

[zxpectre]

I can confirm I have no issues on the db logs @Kammerlo
Do you have Discord? Maybe you can join our server https://dsc.gg/gamechanger and we can have a more fluent chat, even on DMs.

[Kammerlo]

I joined your discord server, we can have a call there today and try to find a solution.
Since there isn't an error message I'm still searching the issue, sorry it takes that long to find the error.