SQL script tests are flaky on CRDB

[tmetzke]

Acceptance Criteria (Required on creation)

• SQL script tests on CRDB don't fail occasionally with comparison failures.

Hints

Stack trace

[2023-07-24T23:25:02.276Z] [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1,102.868 s <<< FAILURE! - in org.camunda.bpm.sql.test.SqlScriptTest

[2023-07-24T23:25:02.276Z] [ERROR] shouldEqualOldUpgradedAndNewCreatedViaLiquibase(org.camunda.bpm.sql.test.SqlScriptTest)  Time elapsed: 178.8 s  <<< FAILURE!

[2023-07-24T23:25:02.277Z] java.lang.AssertionError: Resulting upgraded database misses changes: [createTable tableName=act_ge_bytearray, createTable tableName=act_ge_property, createTable tableName=act_ge_schema_log, createTable tableName=act_hi_actinst, createTable tableName=act_hi_attachment, createTable tableName=act_hi_batch, createTable tableName=act_hi_caseactinst, createTable tableName=act_hi_caseinst, createTable tableName=act_hi_comment, createTable tableName=act_hi_dec_in, createTable tableName=act_hi_dec_out, createTable tableName=act_hi_decinst, createTable tableName=act_hi_detail, createTable tableName=act_hi_ext_task_log, createTable tableName=act_hi_identitylink, createTable tableName=act_hi_incident, createTable tableName=act_hi_job_log, createTable tableName=act_hi_op_log, createTable tableName=act_hi_procinst, createTable tableName=act_hi_taskinst, createTable tableName=act_hi_varinst, createTable tableName=act_id_group, createTable tableName=act_id_info, createTable tableName=act_id_membership, createTable tableName=act_id_tenant, createTable tableName=act_id_tenant_member, createTable tableName=act_id_user, createTable tableName=act_re_camformdef, createTable tableName=act_re_case_def, addUniqueConstraint constraintName=act_hi_caseinst_case_inst_id__key, tableName=act_hi_caseinst, addUniqueConstraint constraintName=act_hi_procinst_proc_inst_id__key, tableName=act_hi_procinst, addUniqueConstraint constraintName=act_uniq_tenant_memb_group, tableName=act_id_tenant_member, addUniqueConstraint constraintName=act_uniq_tenant_memb_user, tableName=act_id_tenant_member, createIndex indexName=act_hi_bat_rm_time, tableName=act_hi_batch, createIndex indexName=act_hi_ext_task_log_proc_def_key, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_procdef, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_procinst, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_rm_time, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_root_pi, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_tenant_id, tableName=act_hi_ext_task_log, createIndex indexName=act_idx_bytear_depl, tableName=act_ge_bytearray, createIndex indexName=act_idx_bytearray_name, tableName=act_ge_bytearray, createIndex indexName=act_idx_bytearray_rm_time, tableName=act_ge_bytearray, createIndex indexName=act_idx_bytearray_root_pi, tableName=act_ge_bytearray, createIndex indexName=act_idx_case_def_tenant_id, tableName=act_re_case_def, createIndex indexName=act_idx_hi_act_inst_comp, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_end, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_proc_def_key, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_procinst, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_rm_time, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_start_end, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_stats, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_tenant_id, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_actinst_root_pi, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_ai_pdefid_end_time, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_attachment_content, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_attachment_procinst, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_attachment_rm_time, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_attachment_root_pi, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_attachment_task, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_cas_a_i_comp, tableName=act_hi_caseactinst, createIndex indexName=act_idx_hi_cas_a_i_create, tableName=act_hi_caseactinst, createIndex indexName=act_idx_hi_cas_a_i_end, tableName=act_hi_caseactinst, createIndex indexName=act_idx_hi_cas_a_i_tenant_id, tableName=act_hi_caseactinst, createIndex indexName=act_idx_hi_cas_i_buskey, tableName=act_hi_caseinst, createIndex indexName=act_idx_hi_cas_i_close, tableName=act_hi_caseinst, createIndex indexName=act_idx_hi_cas_i_tenant_id, tableName=act_hi_caseinst, createIndex indexName=act_idx_hi_casevar_case_inst, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_comment_procinst, tableName=act_hi_comment, createIndex indexName=act_idx_hi_comment_rm_time, tableName=act_hi_comment, createIndex indexName=act_idx_hi_comment_root_pi, tableName=act_hi_comment, createIndex indexName=act_idx_hi_comment_task, tableName=act_hi_comment, createIndex indexName=act_idx_hi_dec_in_clause, tableName=act_hi_dec_in, createIndex indexName=act_idx_hi_dec_in_inst, tableName=act_hi_dec_in, createIndex indexName=act_idx_hi_dec_in_rm_time, tableName=act_hi_dec_in, createIndex indexName=act_idx_hi_dec_in_root_pi, tableName=act_hi_dec_in, createIndex indexName=act_idx_hi_dec_inst_act, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_act_inst, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_ci, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_id, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_key, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_pi, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_req_id, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_req_key, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_rm_time, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_root_id, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_root_pi, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_tenant_id, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_time, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_out_inst, tableName=act_hi_dec_out, createIndex indexName=act_idx_hi_dec_out_rm_time, tableName=act_hi_dec_out, createIndex indexName=act_idx_hi_dec_out_root_pi, tableName=act_hi_dec_out, createIndex indexName=act_idx_hi_dec_out_rule, tableName=act_hi_dec_out, createIndex indexName=act_idx_hi_detail_act_inst, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_bytear, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_case_exec, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_case_inst, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_name, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_proc_def_key, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_proc_inst, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_rm_time, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_root_pi, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_task_bytear, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_task_id, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_tenant_id, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_time, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_var_inst_id, tableName=act_hi_detail, createIndex indexName=act_idx_hi_exttasklog_errordet, tableName=act_hi_ext_task_log, createIndex indexName=act_idx_hi_ident_link_rm_time, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_link_task, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_group, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_proc_def_key, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_root_pi, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_tenant_id, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_timestamp, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_user, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_incident_create_time, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_end_time, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_proc_def_key, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_procinst, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_rm_time, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_root_pi, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_tenant_id, tableName=act_hi_incident, createIndex indexName=act_idx_hi_job_log_ex_stack, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_job_conf, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_job_def_id, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_proc_def_key, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_procdef, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_procinst, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_rm_time, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_root_pi, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_tenant_id, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_op_log_entity_type, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_op_type, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_procdef, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_procinst, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_rm_time, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_root_pi, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_task, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_timestamp, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_user_id, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_pi_pdefid_end_time, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_i_buskey, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_end, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_proc_def_key, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_proc_time, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_rm_time, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_root_pi, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_tenant_id, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_procvar_name_type, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_procvar_proc_inst, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_task_inst_end, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_task_inst_proc_def_key, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_task_inst_rm_time, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_task_inst_start, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_task_inst_tenant_id, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_taskinst_procinst, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_taskinst_root_pi, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_taskinstid_procinst, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_var_inst_proc_def_key, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_var_inst_tenant_id, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_var_pi_name_type, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_act_inst_id, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_bytear, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_name, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_rm_time, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_root_pi, tableName=act_hi_varinst, createIndex indexName=act_idx_memb_group, tableName=act_id_membership, createIndex indexName=act_idx_memb_user, tableName=act_id_membership, createIndex indexName=act_idx_tenant_memb, tableName=act_id_tenant_member, createIndex indexName=act_idx_tenant_memb_group, tableName=act_id_tenant_member, createIndex indexName=act_idx_tenant_memb_user, tableName=act_id_tenant_member, addForeignKeyConstraint baseTableName=act_ru_identitylink, constraintName=act_fk_athrz_procedef, referencedTableName=act_re_procdef, addForeignKeyConstraint baseTableName=act_ru_batch, constraintName=act_fk_batch_job_def, referencedTableName=act_ru_jobdef, addForeignKeyConstraint baseTableName=act_ru_batch, constraintName=act_fk_batch_monitor_job_def, referencedTableName=act_ru_jobdef, addForeignKeyConstraint baseTableName=act_ru_batch, constraintName=act_fk_batch_seed_job_def, referencedTableName=act_ru_jobdef, addForeignKeyConstraint baseTableName=act_ge_bytearray, constraintName=act_fk_bytearr_depl, referencedTableName=act_re_deployment, addForeignKeyConstraint baseTableName=act_ru_case_execution, constraintName=act_fk_case_exe_case_def, referencedTableName=act_re_case_def, addForeignKeyConstraint baseTableName=act_ru_case_execution, constraintName=act_fk_case_exe_case_inst, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_case_execution, constraintName=act_fk_case_exe_parent, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_case_sentry_part, constraintName=act_fk_case_sentry_case_exec, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_case_sentry_part, constraintName=act_fk_case_sentry_case_inst, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_re_decision_def, constraintName=act_fk_dec_req, referencedTableName=act_re_decision_req_def, addForeignKeyConstraint baseTableName=act_ru_event_subscr, constraintName=act_fk_event_exec, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_execution, constraintName=act_fk_exe_parent, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_execution, constraintName=act_fk_exe_procdef, referencedTableName=act_re_procdef, addForeignKeyConstraint baseTableName=act_ru_execution, constraintName=act_fk_exe_procinst, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_execution, constraintName=act_fk_exe_super, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_ext_task, constraintName=act_fk_ext_task_error_details, referencedTableName=act_ge_bytearray, addForeignKeyConstraint baseTableName=act_ru_ext_task, constraintName=act_fk_ext_task_exe, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_cause, referencedTableName=act_ru_incident, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_exe, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_job_def, referencedTableName=act_ru_jobdef, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_procdef, referencedTableName=act_re_procdef, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_procinst, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_rcause, referencedTableName=act_ru_incident, addForeignKeyConstraint baseTableName=act_ru_job, constraintName=act_fk_job_exception, referencedTableName=act_ge_bytearray, addForeignKeyConstraint baseTableName=act_id_membership, constraintName=act_fk_memb_group, referencedTableName=act_id_group, addForeignKeyConstraint baseTableName=act_id_membership, constraintName=act_fk_memb_user, referencedTableName=act_id_user, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_case_def, referencedTableName=act_re_case_def, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_case_exe, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_exe, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_procdef, referencedTableName=act_re_procdef, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_procinst, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_id_tenant_member, constraintName=act_fk_tenant_memb, referencedTableName=act_id_tenant, addForeignKeyConstraint baseTableName=act_id_tenant_member, constraintName=act_fk_tenant_memb_group, referencedTableName=act_id_group, addForeignKeyConstraint baseTableName=act_id_tenant_member, constraintName=act_fk_tenant_memb_user, referencedTableName=act_id_user, addForeignKeyConstraint baseTableName=act_ru_identitylink, constraintName=act_fk_tskass_task, referencedTableName=act_ru_task, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_batch, referencedTableName=act_ru_batch, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_bytearray, referencedTableName=act_ge_bytearray, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_case_exe, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_case_inst, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_exe, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_procinst, referencedTableName=act_ru_execution]

[2023-07-24T23:25:02.280Z] 	at org.camunda.bpm.sql.test.SqlScriptTest.shouldEqualOldUpgradedAndNewCreatedViaLiquibase(SqlScriptTest.java:156)

[2023-07-24T23:25:02.280Z] 

[2023-07-24T23:25:02.582Z] [INFO] 

[2023-07-24T23:25:02.582Z] [INFO] Results:

[2023-07-24T23:25:02.582Z] [INFO] 

[2023-07-24T23:25:02.582Z] [ERROR] Failures: 

[2023-07-24T23:25:02.582Z] [ERROR]   SqlScriptTest.shouldEqualOldUpgradedAndNewCreatedViaLiquibase:156 Resulting upgraded database misses changes: [createTable tableName=act_ge_bytearray, createTable tableName=act_ge_property, createTable tableName=act_ge_schema_log, createTable tableName=act_hi_actinst, createTable tableName=act_hi_attachment, createTable tableName=act_hi_batch, createTable tableName=act_hi_caseactinst, createTable tableName=act_hi_caseinst, createTable tableName=act_hi_comment, createTable tableName=act_hi_dec_in, createTable tableName=act_hi_dec_out, createTable tableName=act_hi_decinst, createTable tableName=act_hi_detail, createTable tableName=act_hi_ext_task_log, createTable tableName=act_hi_identitylink, createTable tableName=act_hi_incident, createTable tableName=act_hi_job_log, createTable tableName=act_hi_op_log, createTable tableName=act_hi_procinst, createTable tableName=act_hi_taskinst, createTable tableName=act_hi_varinst, createTable tableName=act_id_group, createTable tableName=act_id_info, createTable tableName=act_id_membership, createTable tableName=act_id_tenant, createTable tableName=act_id_tenant_member, createTable tableName=act_id_user, createTable tableName=act_re_camformdef, createTable tableName=act_re_case_def, addUniqueConstraint constraintName=act_hi_caseinst_case_inst_id__key, tableName=act_hi_caseinst, addUniqueConstraint constraintName=act_hi_procinst_proc_inst_id__key, tableName=act_hi_procinst, addUniqueConstraint constraintName=act_uniq_tenant_memb_group, tableName=act_id_tenant_member, addUniqueConstraint constraintName=act_uniq_tenant_memb_user, tableName=act_id_tenant_member, createIndex indexName=act_hi_bat_rm_time, tableName=act_hi_batch, createIndex indexName=act_hi_ext_task_log_proc_def_key, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_procdef, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_procinst, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_rm_time, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_root_pi, tableName=act_hi_ext_task_log, createIndex indexName=act_hi_ext_task_log_tenant_id, tableName=act_hi_ext_task_log, createIndex indexName=act_idx_bytear_depl, tableName=act_ge_bytearray, createIndex indexName=act_idx_bytearray_name, tableName=act_ge_bytearray, createIndex indexName=act_idx_bytearray_rm_time, tableName=act_ge_bytearray, createIndex indexName=act_idx_bytearray_root_pi, tableName=act_ge_bytearray, createIndex indexName=act_idx_case_def_tenant_id, tableName=act_re_case_def, createIndex indexName=act_idx_hi_act_inst_comp, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_end, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_proc_def_key, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_procinst, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_rm_time, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_start_end, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_stats, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_act_inst_tenant_id, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_actinst_root_pi, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_ai_pdefid_end_time, tableName=act_hi_actinst, createIndex indexName=act_idx_hi_attachment_content, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_attachment_procinst, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_attachment_rm_time, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_attachment_root_pi, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_attachment_task, tableName=act_hi_attachment, createIndex indexName=act_idx_hi_cas_a_i_comp, tableName=act_hi_caseactinst, createIndex indexName=act_idx_hi_cas_a_i_create, tableName=act_hi_caseactinst, createIndex indexName=act_idx_hi_cas_a_i_end, tableName=act_hi_caseactinst, createIndex indexName=act_idx_hi_cas_a_i_tenant_id, tableName=act_hi_caseactinst, createIndex indexName=act_idx_hi_cas_i_buskey, tableName=act_hi_caseinst, createIndex indexName=act_idx_hi_cas_i_close, tableName=act_hi_caseinst, createIndex indexName=act_idx_hi_cas_i_tenant_id, tableName=act_hi_caseinst, createIndex indexName=act_idx_hi_casevar_case_inst, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_comment_procinst, tableName=act_hi_comment, createIndex indexName=act_idx_hi_comment_rm_time, tableName=act_hi_comment, createIndex indexName=act_idx_hi_comment_root_pi, tableName=act_hi_comment, createIndex indexName=act_idx_hi_comment_task, tableName=act_hi_comment, createIndex indexName=act_idx_hi_dec_in_clause, tableName=act_hi_dec_in, createIndex indexName=act_idx_hi_dec_in_inst, tableName=act_hi_dec_in, createIndex indexName=act_idx_hi_dec_in_rm_time, tableName=act_hi_dec_in, createIndex indexName=act_idx_hi_dec_in_root_pi, tableName=act_hi_dec_in, createIndex indexName=act_idx_hi_dec_inst_act, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_act_inst, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_ci, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_id, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_key, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_pi, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_req_id, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_req_key, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_rm_time, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_root_id, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_root_pi, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_tenant_id, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_inst_time, tableName=act_hi_decinst, createIndex indexName=act_idx_hi_dec_out_inst, tableName=act_hi_dec_out, createIndex indexName=act_idx_hi_dec_out_rm_time, tableName=act_hi_dec_out, createIndex indexName=act_idx_hi_dec_out_root_pi, tableName=act_hi_dec_out, createIndex indexName=act_idx_hi_dec_out_rule, tableName=act_hi_dec_out, createIndex indexName=act_idx_hi_detail_act_inst, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_bytear, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_case_exec, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_case_inst, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_name, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_proc_def_key, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_proc_inst, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_rm_time, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_root_pi, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_task_bytear, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_task_id, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_tenant_id, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_time, tableName=act_hi_detail, createIndex indexName=act_idx_hi_detail_var_inst_id, tableName=act_hi_detail, createIndex indexName=act_idx_hi_exttasklog_errordet, tableName=act_hi_ext_task_log, createIndex indexName=act_idx_hi_ident_link_rm_time, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_link_task, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_group, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_proc_def_key, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_root_pi, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_tenant_id, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_timestamp, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_ident_lnk_user, tableName=act_hi_identitylink, createIndex indexName=act_idx_hi_incident_create_time, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_end_time, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_proc_def_key, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_procinst, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_rm_time, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_root_pi, tableName=act_hi_incident, createIndex indexName=act_idx_hi_incident_tenant_id, tableName=act_hi_incident, createIndex indexName=act_idx_hi_job_log_ex_stack, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_job_conf, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_job_def_id, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_proc_def_key, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_procdef, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_procinst, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_rm_time, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_root_pi, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_job_log_tenant_id, tableName=act_hi_job_log, createIndex indexName=act_idx_hi_op_log_entity_type, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_op_type, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_procdef, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_procinst, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_rm_time, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_root_pi, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_task, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_timestamp, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_op_log_user_id, tableName=act_hi_op_log, createIndex indexName=act_idx_hi_pi_pdefid_end_time, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_i_buskey, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_end, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_proc_def_key, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_proc_time, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_rm_time, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_root_pi, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_pro_inst_tenant_id, tableName=act_hi_procinst, createIndex indexName=act_idx_hi_procvar_name_type, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_procvar_proc_inst, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_task_inst_end, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_task_inst_proc_def_key, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_task_inst_rm_time, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_task_inst_start, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_task_inst_tenant_id, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_taskinst_procinst, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_taskinst_root_pi, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_taskinstid_procinst, tableName=act_hi_taskinst, createIndex indexName=act_idx_hi_var_inst_proc_def_key, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_var_inst_tenant_id, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_var_pi_name_type, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_act_inst_id, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_bytear, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_name, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_rm_time, tableName=act_hi_varinst, createIndex indexName=act_idx_hi_varinst_root_pi, tableName=act_hi_varinst, createIndex indexName=act_idx_memb_group, tableName=act_id_membership, createIndex indexName=act_idx_memb_user, tableName=act_id_membership, createIndex indexName=act_idx_tenant_memb, tableName=act_id_tenant_member, createIndex indexName=act_idx_tenant_memb_group, tableName=act_id_tenant_member, createIndex indexName=act_idx_tenant_memb_user, tableName=act_id_tenant_member, addForeignKeyConstraint baseTableName=act_ru_identitylink, constraintName=act_fk_athrz_procedef, referencedTableName=act_re_procdef, addForeignKeyConstraint baseTableName=act_ru_batch, constraintName=act_fk_batch_job_def, referencedTableName=act_ru_jobdef, addForeignKeyConstraint baseTableName=act_ru_batch, constraintName=act_fk_batch_monitor_job_def, referencedTableName=act_ru_jobdef, addForeignKeyConstraint baseTableName=act_ru_batch, constraintName=act_fk_batch_seed_job_def, referencedTableName=act_ru_jobdef, addForeignKeyConstraint baseTableName=act_ge_bytearray, constraintName=act_fk_bytearr_depl, referencedTableName=act_re_deployment, addForeignKeyConstraint baseTableName=act_ru_case_execution, constraintName=act_fk_case_exe_case_def, referencedTableName=act_re_case_def, addForeignKeyConstraint baseTableName=act_ru_case_execution, constraintName=act_fk_case_exe_case_inst, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_case_execution, constraintName=act_fk_case_exe_parent, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_case_sentry_part, constraintName=act_fk_case_sentry_case_exec, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_case_sentry_part, constraintName=act_fk_case_sentry_case_inst, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_re_decision_def, constraintName=act_fk_dec_req, referencedTableName=act_re_decision_req_def, addForeignKeyConstraint baseTableName=act_ru_event_subscr, constraintName=act_fk_event_exec, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_execution, constraintName=act_fk_exe_parent, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_execution, constraintName=act_fk_exe_procdef, referencedTableName=act_re_procdef, addForeignKeyConstraint baseTableName=act_ru_execution, constraintName=act_fk_exe_procinst, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_execution, constraintName=act_fk_exe_super, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_ext_task, constraintName=act_fk_ext_task_error_details, referencedTableName=act_ge_bytearray, addForeignKeyConstraint baseTableName=act_ru_ext_task, constraintName=act_fk_ext_task_exe, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_cause, referencedTableName=act_ru_incident, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_exe, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_job_def, referencedTableName=act_ru_jobdef, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_procdef, referencedTableName=act_re_procdef, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_procinst, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_incident, constraintName=act_fk_inc_rcause, referencedTableName=act_ru_incident, addForeignKeyConstraint baseTableName=act_ru_job, constraintName=act_fk_job_exception, referencedTableName=act_ge_bytearray, addForeignKeyConstraint baseTableName=act_id_membership, constraintName=act_fk_memb_group, referencedTableName=act_id_group, addForeignKeyConstraint baseTableName=act_id_membership, constraintName=act_fk_memb_user, referencedTableName=act_id_user, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_case_def, referencedTableName=act_re_case_def, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_case_exe, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_exe, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_procdef, referencedTableName=act_re_procdef, addForeignKeyConstraint baseTableName=act_ru_task, constraintName=act_fk_task_procinst, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_id_tenant_member, constraintName=act_fk_tenant_memb, referencedTableName=act_id_tenant, addForeignKeyConstraint baseTableName=act_id_tenant_member, constraintName=act_fk_tenant_memb_group, referencedTableName=act_id_group, addForeignKeyConstraint baseTableName=act_id_tenant_member, constraintName=act_fk_tenant_memb_user, referencedTableName=act_id_user, addForeignKeyConstraint baseTableName=act_ru_identitylink, constraintName=act_fk_tskass_task, referencedTableName=act_ru_task, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_batch, referencedTableName=act_ru_batch, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_bytearray, referencedTableName=act_ge_bytearray, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_case_exe, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_case_inst, referencedTableName=act_ru_case_execution, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_exe, referencedTableName=act_ru_execution, addForeignKeyConstraint baseTableName=act_ru_variable, constraintName=act_fk_var_procinst, referencedTableName=act_ru_execution]

Links

• 🔒 Failing build

Breakdown

Pull Requests

Beta Give feedback

1. test(sql-script): stabilize cleanup and comparison #3625
   ci:all-db +1
   
2. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1067
3. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1068
4. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1069

[tmetzke]

Root causing

• On CRDB (and on very few occasions also on other databases), the database is not always cleaned up completely by the Liquibase#dropAll command. The DATABASECHANGELOG table is not deleted/cleaned up, thus still containing the changeset entries. The Camunda-specific tables are removed. This has been reported in the Liquibase repository as well already: dropAll sometimes does not delete all tables liquibase/liquibase#2851. The exact root cause is unknown.
• When the Liquibase#update command is invoked, the changesets are not executed because they are still present in the DATABASECHANGELOG.
• The database now contains no Camunda-specific tables and the diffing yields all (sometimes also only a subset) of the Camunda tables. This leads to the failed assertion mentioned in the description.

Solution options

1. Clean up the tables manually by issuing DELETE statements (Camunda-provided and custom ones for Liquibase tables).
2. Invoke the dropAll command at least twice, reducing the potential for an unclean database state drastically.

Decisions

• Since the manual cleanup led to deadlocks on several databases, I decided to move along with option 2, invoking the dropAll twice in cleanup procedures.

[tmetzke]

To reviewer
Backports are exact cherry-picks, no adjustments.

[yanavasileva]

📓 Don't forget to add the labels for the backports.