Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transaction killed while uploading a run #1976

Open
lampajr opened this issue Sep 11, 2024 · 0 comments · May be fixed by #2172
Open

Transaction killed while uploading a run #1976

lampajr opened this issue Sep 11, 2024 · 0 comments · May be fixed by #2172

Comments

@lampajr
Copy link
Member

lampajr commented Sep 11, 2024

Describe the bug

Looks like that trying to upload a run for some specific tests having a lot of dataset (10k or more) makes the active transaction to be forcefully killed (did we hit a timeout?)

2024-09-09 17:16:26,946 797ee630d871 quarkus-run.jar[7] WARN  [com.net.sch.UnknownKeywordFactory] (executor-thread-17) Unknown keyword $id - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
2024-09-09 17:18:15,573 797ee630d871 quarkus-run.jar[7] WARN  [com.arj.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check processing TX 0:ffff0a000264:9bc3:66df1d30:104 in state  RUN
2024-09-09 17:18:15,575 797ee630d871 quarkus-run.jar[7] WARN  [com.arj.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffff0a000264:9bc3:66df1d30:104 invoked while multiple threads active within it.
2024-09-09 17:18:15,576 797ee630d871 quarkus-run.jar[7] WARN  [com.arj.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012381: Action id 0:ffff0a000264:9bc3:66df1d30:104 completed with multiple threads - thread executor-thread-17 was in progress with [email protected]/sun.nio.ch.Net.poll(Native Method)
[email protected]/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:186)
[email protected]/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:195)
...
org.hibernate.query.spi.AbstractQuery.executeUpdate(AbstractQuery.java:651)
io.hyperfoil.tools.horreum.svc.DatasetServiceImpl.calculateLabelValues(DatasetServiceImpl.java:453)
io.hyperfoil.tools.horreum.svc.DatasetServiceImpl_Subclass.calculateLabelValues$$superforward(Unknown Source)
io.hyperfoil.tools.horreum.svc.DatasetServiceImpl_Subclass$$function$$6.apply(Unknown Source)

It seems the transaction got killed while calling a postgres procedure

em.createNativeQuery("call calc_dataset_view(?1);").setParameter(1, datasetId).executeUpdate();
:

2024-09-09 17:18:15,576 797ee630d871 quarkus-run.jar[7] WARN  [com.arj.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff0a000264:9bc3:66df1d30:104 aborting with 1 threads active!
2024-09-09 17:18:15,585 797ee630d871 quarkus-run.jar[7] WARN  [org.hib.eng.jdb.spi.SqlExceptionHelper] (executor-thread-17) SQL Error: 0, SQLState: 57014
2024-09-09 17:18:15,585 797ee630d871 quarkus-run.jar[7] ERROR [org.hib.eng.jdb.spi.SqlExceptionHelper] (executor-thread-17) ERROR: canceling statement due to user request
  Where: SQL statement "WITH view_agg AS (
                SELECT
                        vc.view_id, vc.id as vcid, array_agg(DISTINCT label.id) as label_ids, jsonb_object_agg(label.name, lv.value) as value FROM dataset_schemas ds
                JOIN label ON label.schema_id = ds.schema_id
                JOIN viewcomponent vc ON vc.labels ? label.name
                JOIN label_values lv ON lv.label_id = label.id
                WHERE ds.dataset_id = datasetId
                GROUP BY vc.view_id, vcid
            )
            INSERT INTO dataset_view (dataset_id, view_id, label_ids, value)
            SELECT datasetId, view_id, array_agg(DISTINCT label_id), jsonb_object_agg(vcid, value) FROM view_agg, unnest(label_ids) as label_id
            GROUP BY view_id"
PL/pgSQL function calc_dataset_view(bigint) line 3 at SQL statement

And this force the run upload to fail:

2024-09-09 17:18:15,586 797ee630d871 quarkus-run.jar[7] WARN  [org.hib.res.tra.bac.jta.int.syn.SynchronizationCallbackCoordinatorTrackingImpl] (Transaction Reaper Worker 0) HHH000451: Transaction afterCompletion called by a background thread; delaying afterCompletion processing until the original thread can handle it. [status=4]
2024-09-09 17:18:15,586 797ee630d871 quarkus-run.jar[7] WARN  [com.arj.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a000264:9bc3:66df1d30:104
2024-09-09 17:18:15,589 797ee630d871 quarkus-run.jar[7] WARN  [com.arj.ats.arjuna] (executor-thread-17) ARJUNA012077: Abort called on already aborted atomic action 0:ffff0a000264:9bc3:66df1d30:104
2024-09-09 17:18:15,592 797ee630d871 quarkus-run.jar[7] ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-17) HTTP Request to /api/run/data?start=%24.timestamp&stop=%24.timestamp&test=MY_AWESOME_TEST&owner=my-awesome-team&access=PUBLIC&schema=urn%3Akie-benchmarks%3A0.2 failed, error id: dd652125-ef2b-42d7-8bdb-09169b21f20d-1: io.quarkus.arc.ArcUndeclaredThrowableException: Error invoking subclass method
	at io.hyperfoil.tools.horreum.svc.RunServiceImpl_Subclass.addRunFromData(Unknown Source)
	at io.hyperfoil.tools.horreum.svc.RunServiceImpl.addRunFromData(RunServiceImpl.java:538)

To Reproduce

Was not able to reproduce the same locally yet.

Version

What is the version of Horreum ?

If you are using a development branch; what is the commit id ?

0.14.1
@lampajr lampajr linked a pull request Nov 15, 2024 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant