Incremental changelog producing broken changelogs when adding/removing fields and relationships in the same jdl run

[OmarHawk]

Overview of the issue

We are using the jhipster jdl command in combination with an application that was generated using the --incremental-changelog option. When we add and/or remove fields and in the same jdl run add a new relationship, the liquibase changelogs being created misses out the field changes at all and strange changelog files not fitting to the actual data model are being produced.

Motivation for or Use Case

Incremental changelog should produce consistent database changes.

Reproduce the error

1. We have created a general application with activated incremental-changelog option jhipster --incremental-changelog and answered the questions according to the attached config.
2. We apply the following jdl. jhipster jdl jhipster-jdl01.jdl It generates a proper data model.

entity EntityA {
   active Boolean
   status String maxlength(30)
   @Id uuid UUID
   sourceid Long
   someDate Instant
}

entity StreamLog {
   comment String
   status String
   date Instant
}

// Set pagination options
paginate all with pagination

// Use Data Transfer Objects (DTO)
dto all with mapstruct

// Set service options to all
service all with serviceImpl

// filtering support
filter all

3. Now, we apply the jdl with the following contents (notice the added anotherBoolean field in EntityA and the removed comment field in StreamLog + the newly added commentNew field and also the new relationship). jhipster jdl jhipster-jdl02.jdl

entity EntityA {
   active Boolean
   status String maxlength(30)
   @Id uuid UUID
   sourceid Long
   someDate Instant
   anotherBoolean Boolean
}

entity StreamLog {
   commentNew String
   status String
   date Instant
}

relationship ManyToOne {
   StreamLog{entityA}
      to
   EntityA{streamLog}
}

// Set pagination options
paginate all with pagination

// Use Data Transfer Objects (DTO)
dto all with mapstruct

// Set service options to all
service all with serviceImpl

// filtering support
filter all

4. It generates changelog files that look like the following. As you can see, it generates twice the same changelog files and changes for StreamLog entity, but none for EntityA:
   
   
5. If you take a look into both StreamLog change files ~'updated_entity_StreamLog.xml', you'll see that it missed to generate the dropColumn / addColumn statements for the comment vs. commentNew change
   

Related issues

Nothing found.

Suggest a Fix

I analyzed this and I believe, I found the root cause of this already, but not sure, how to efficiently solve this.

We do have the prepareChangeog call twice, once for field changes here:

generator-jhipster/generators/liquibase/generator.ts

Lines 216 to 228 in 62f057d

	this.databaseChangelogs.push(
	this.prepareChangelog({
	databaseChangelog: {
	...databaseChangelog,
	fieldChangelog: true,
	addedRelationships: [],
	removedRelationships: [],
	relationshipsToRecreateForeignKeysOnly: [],
	},
	application,
	}),
	);
	}

and once for the relationship changes a little further down below, here:

generator-jhipster/generators/liquibase/generator.ts

Lines 236 to 248 in 62f057d

	this.databaseChangelogs.push(
	this.prepareChangelog({
	databaseChangelog: {
	...databaseChangelog,
	relationshipChangelog: true,
	addedFields: [],
	removedFields: [],
	},
	application,
	}),
	);
	}
	}

Both use the spread operator to create a shallow copy and set a few fields to empty array on the changelog to distinguish these kind of changelogs.

In the run for the field changes, everything is kind of alright still, as it will just figure out the added/removed fields.
In the run for the relationship changes, an empty array is passed for addedFields / removedFields.
In both prepareChangelog runs, it operates on the reference to changelogData:

generator-jhipster/generators/liquibase/generator.ts

Line 607 in 62f057d

const entityChanges = databaseChangelog.changelogData;

So in the else here:

generator-jhipster/generators/liquibase/generator.ts

Lines 612 to 617 in 62f057d

	if (databaseChangelog.newEntity) {
	entityChanges.fields = entityChanges.allFields;
	} else {
	entityChanges.addedFields = databaseChangelog.addedFields.filter(field => !field.transient);
	entityChanges.removedFields = databaseChangelog.removedFields.filter(field => !field.transient);
	}

It does not only set the addedFields / removedFields on the relationship changelog to empty array (as it was passed), but also in the already calculated other changelogs for the field changes, clearing the changelogData.addedFields / changelogData.removedFields on the field changelog.

JHipster Version(s)

Latest master. 8.2.1. (I have been working on a Pull Request and noticed it thereby)

JHipster configuration, a .yo-rc.json file generated in the root folder

.yo-rc.json file

{
  "generator-jhipster": {
    "applicationType": "monolith",
    "authenticationType": "session",
    "baseName": "defFields",
    "buildTool": "maven",
    "cacheProvider": "no",
    "clientFramework": "angular",
    "clientTestFrameworks": [],
    "clientTheme": "none",
    "creationTimestamp": 1710952592339,
    "databaseType": "sql",
    "devDatabaseType": "mysql",
    "devServerPort": 4200,
    "enableGradleEnterprise": null,
    "enableHibernateCache": false,
    "enableSwaggerCodegen": false,
    "enableTranslation": true,
    "entities": [
      "EntityA",
      "StreamLog"
    ],
    "feignClient": null,
    "gradleEnterpriseHost": null,
    "incrementalChangelog": true,
    "jhipsterVersion": "8.2.1",
    "languages": [
      "de"
    ],
    "lastLiquibaseTimestamp": 1710952848000,
    "messageBroker": false,
    "microfrontend": null,
    "microfrontends": [],
    "nativeLanguage": "de",
    "packageName": "com.mycompany.myapp",
    "prodDatabaseType": "mysql",
    "reactive": false,
    "searchEngine": false,
    "serverPort": null,
    "serverSideOptions": [],
    "serviceDiscoveryType": false,
    "syncUserWithIdp": null,
    "testFrameworks": [],
    "websocket": false,
    "withAdminUi": true
  }
}

Environment and Tools

openjdk version "17.0.7" 2023-04-18 LTS
OpenJDK Runtime Environment (Red_Hat-17.0.7.0+7-1) (build 17.0.7+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.7.0+7-1) (build 17.0.7+7-LTS, mixed mode, sharing)

git version 2.44.0.windows.1

node: v18.19.0
npm: 10.2.3

Docker version 24.0.7-rd, build 72ffacf

JDL for the Entity configuration(s) entityName.json files generated in the .jhipster directory

JDL entity definitions

@ChangelogDate("20240320163854")
entity EntityA (entity_a) {
  active Boolean
  status String maxlength(30)
  uuid UUID
  sourceid Long
  someDate Instant
  anotherBoolean Boolean
}
@ChangelogDate("20240320163855")
entity StreamLog {
  commentNew String
  status String
  date Instant
}
relationship ManyToOne {
  StreamLog{entityA} to EntityA{streamLog}
}

dto EntityA, StreamLog with mapstruct
paginate EntityA, StreamLog with pagination
service EntityA, StreamLog with serviceImpl
search EntityA, StreamLog with no
filter EntityA, StreamLog

Entity configuration(s) entityName.json files generated in the .jhipster directory

These are the final .json files after the last step:

{
  "annotations": {
    "changelogDate": "20240320163854"
  },
  "applications": "*",
  "dto": "mapstruct",
  "entityTableName": "entity_a",
  "fields": [
    {
      "fieldName": "active",
      "fieldType": "Boolean"
    },
    {
      "fieldName": "status",
      "fieldType": "String",
      "fieldValidateRules": ["maxlength"],
      "fieldValidateRulesMaxlength": "30"
    },
    {
      "fieldName": "uuid",
      "fieldType": "UUID",
      "options": {
        "id": true
      }
    },
    {
      "fieldName": "sourceid",
      "fieldType": "Long"
    },
    {
      "fieldName": "someDate",
      "fieldType": "Instant"
    },
    {
      "fieldName": "anotherBoolean",
      "fieldType": "Boolean"
    }
  ],
  "incrementalChangelog": true,
  "jpaMetamodelFiltering": true,
  "name": "EntityA",
  "pagination": "pagination",
  "relationships": [
    {
      "otherEntityName": "streamLog",
      "otherEntityRelationshipName": "entityA",
      "relationshipName": "streamLog",
      "relationshipSide": "right",
      "relationshipType": "one-to-many"
    }
  ],
  "searchEngine": "no",
  "service": "serviceImpl"
}

{
  "annotations": {
    "changelogDate": "20240320163855"
  },
  "applications": "*",
  "dto": "mapstruct",
  "fields": [
    {
      "fieldName": "commentNew",
      "fieldType": "String"
    },
    {
      "fieldName": "status",
      "fieldType": "String"
    },
    {
      "fieldName": "date",
      "fieldType": "Instant"
    }
  ],
  "incrementalChangelog": true,
  "jpaMetamodelFiltering": true,
  "name": "StreamLog",
  "pagination": "pagination",
  "relationships": [
    {
      "otherEntityName": "entityA",
      "otherEntityRelationshipName": "streamLog",
      "relationshipName": "entityA",
      "relationshipSide": "left",
      "relationshipType": "many-to-one"
    }
  ],
  "searchEngine": "no",
  "service": "serviceImpl"
}

Browsers and Operating System

• Checking this box is mandatory (this is just to show you read everything)