Crawler 10.5.0 fails to deploy

[skarampatakis]

Description - What happened? *

Hi, we are running AP 4.1.4 and we are now trying to deploy the catalog-crawler. According to the release notes, this should be on version 10.5.0.
I am deploying the crawler through docker, but when the container starts, I am getting some error in regards of the database migrations, please see logs.

I am not familiar with flyway so I am not sure how to fix this. Shouldn't those migrations happen automatically?

Please let me know how to proceed with this.

Expected Behavior *

The crawler deploys and it is able to crawl the connectors registered on the specified environment.

Observed Behavior *

The crawler fails to deploy

Steps to Reproduce

No response

Context Information

No response

Relevant log output

Bad level value for property: org.eclipse.edc.api.observability.ObservabilityApiController.level
2025-01-17 18:01:31 Configuration file does not exist: dataspaceconnector-configuration.properties. Ignoring. 
2025-01-17 18:01:31 Initialized FS Configuration 
2025-01-17 18:01:31 Initialized Boot Services 
2025-01-17 18:01:31 Initialized JSON-LD Extension 
2025-01-17 18:01:31 Initialized FS Vault 
2025-01-17 18:01:31 Initialized Core Default Services 
2025-01-17 18:01:31 HTTPS enforcement it not enabled, please enable it in a production environment 
2025-01-17 18:01:31 HTTPS enforcement it not enabled, please enable it in a production environment 
2025-01-17 18:01:31 Initialized Core Services 
2025-01-17 18:01:31 Initialized Control Plane Default Services 
2025-01-17 18:01:31 Initialized Jetty Service 
2025-01-17 18:01:31 Initialized Jersey Web Service 
2025-01-17 18:01:31 Protocol API will be available under port=11003, path=/api/dsp 
2025-01-17 18:01:31 Initialized Dataspace Protocol API Configuration Extension 
2025-01-17 18:01:31 Initialized Transfer Process Default Services 
2025-01-17 18:01:31 Initialized org.eclipse.edc.connector.contract.ContractNegotiationDefaultServicesExtension 
2025-01-17 18:01:31 Initialized Transfer Core 
2025-01-17 18:01:31 Initialized org.eclipse.edc.connector.transfer.TransferProcessCommandExtension 
2025-01-17 18:01:31 Initialized Contract Core 
2025-01-17 18:01:31 Initialized Contract Negotiation command handlers 
2025-01-17 18:01:31 Initialized Observability API 
2025-01-17 18:01:31 Initialized Data Plane Selector Default Services 
2025-01-17 18:01:31 Initialized Catalog Default Services 
2025-01-17 18:01:31 Initialized Catalog Core 
2025-01-17 18:01:31 No TransactionContext registered, a no-op implementation will be used, not suitable for production environments 
2025-01-17 18:01:31 Initialized Control Plane Services 
2025-01-17 18:01:31 Initialized Api Core Default Services 
2025-01-17 18:01:31 No AuthenticationService registered, an all-pass implementation will be used, not suitable for production environments 
2025-01-17 18:01:31 Management API will be available under port=11002, path=/api/management 
2025-01-17 18:01:31 Initialized Management API configuration 
2025-01-17 18:01:32 Flyway migration locations for 'flyway_schema_history': [classpath:db-crawler/migration] 
2025-01-17 18:01:32 Error booting runtime: Validate failed: Migrations have failed validation Detected resolved migration not applied to database: 1. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 2. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 3. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 4. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 5. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 6. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 7. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 8. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 9. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 10. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 12. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE 
org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation
Detected resolved migration not applied to database: 1.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 2.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 3.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 4.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 5.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 6.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 7.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 8.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 9.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 10.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 12.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE
        at org.flywaydb.core.Flyway$4.execute(Flyway.java:252)
        at org.flywaydb.core.Flyway$4.execute(Flyway.java:242)
        at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:204)
        at org.flywaydb.core.Flyway.validate(Flyway.java:242)
        at de.sovity.edc.extension.postgresql.FlywayUtils.validate(FlywayUtils.java:65)
        at de.sovity.edc.extension.postgresql.FlywayUtils.migrateOrValidate(FlywayUtils.java:59)
        at de.sovity.edc.extension.postgresql.FlywayUtils.cleanAndMigrate(FlywayUtils.java:37)
        at de.sovity.edc.ext.catalog.crawler.dao.config.FlywayService.validateOrMigrateInTests(FlywayService.java:42)
        at de.sovity.edc.ext.catalog.crawler.CrawlerExtensionContextBuilder.buildContext(CrawlerExtensionContextBuilder.java:116)
        at de.sovity.edc.ext.catalog.crawler.CrawlerExtension.initialize(CrawlerExtension.java:120)
        at org.eclipse.edc.boot.system.injection.lifecycle.InitializePhase.initialize(InitializePhase.java:37)
        at org.eclipse.edc.boot.system.injection.lifecycle.ExtensionLifecycleManager.initialize(ExtensionLifecycleManager.java:61)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
        at org.eclipse.edc.boot.system.ExtensionLoader.bootServiceExtensions(ExtensionLoader.java:67)
        at org.eclipse.edc.boot.system.runtime.BaseRuntime.bootExtensions(BaseRuntime.java:141)
        at org.eclipse.edc.boot.system.runtime.BaseRuntime.boot(BaseRuntime.java:202)
        at org.eclipse.edc.boot.system.runtime.BaseRuntime.boot(BaseRuntime.java:84)
        at org.eclipse.edc.boot.system.runtime.BaseRuntime.main(BaseRuntime.java:72)

Screenshots

No response

[VladimirAlexiev]

@ivanov-petar can you please contact Sovity because we need this ASAP?

[tmberthold]

I'm not an expert on Flyway database migrations, but after internal consultation the log suggests the following:

Error booting runtime: Validate failed: Migrations have failed validation Detected resolved migration not applied to database

The error may result from a corrupted database state:

• The Flyway history table indicates that all migrations were successfully applied ("resolved migration")
• However, the actual database schema seems inconsistent with the expected state post-migration ("not applied to database")

This type of issue can arise if the database state was unintentionally corrupted, potentially due to external factors or incomplete migrations beforehand. Could you please try with a fresh new setup of databases?

[skarampatakis]

Hi Tim,
I actually tried the deployment locally on an empty database, is this considered as a new setup of the database?
Shall I do some steps before or run some scripts?

[kamilczaja]

Hi Tim, I actually tried the deployment locally on an empty database, is this considered as a new setup of the database? Shall I do some steps before or run some scripts?

Hi!
What do you mean with "empty database"? Is it completely empty? Does the table flyway_schema_history exist and if yes, does it have any data?

Are you sure the Catalog Crawler is pointing to the correct database?

Are you trying to start the Crawler without having the actual portal backend running?

[skarampatakis]

I am deploying the crawler on a local instance to test if it starts up before we deploy on our AP, so I only have the container of the crawler and an empty PostgreSQL database. So there are not tables existing. So I think my answer to all your questions is yes.

Shall I deploy additional components to test it properly? (the full AP?)

I am kind of afraid on deploying it directly on our working environment.

[kamilczaja]

Ah that explains the problem.
The Crawler validates database migrations but does not migrate on its own. It expects the database to be in an already set up and migrated state.

To mitigate this, start up the Authority Portal Backend connected to the same database (local is fine, as long as it's the same DB) to let it create all necessary tables and go through the migrations and then start up the Crawler. It should then validate the DB schema successfully.

[skarampatakis]

Thanks @kamilczaja , so just to be sure, only the backend is enough, or all of its other dependencies, other than postgres, should be there as well (KC, DAPS-KC)?

Would it work if I import a dump of our database on my local empty database? Perhaps this way would be easier to see if it actually works, my expectation would be to see some crawling in action.

Please excuse me for any "silly" questions.

[kamilczaja]

Hey,

Backend + Postgres (potentially also IAM Keycloak) should be enough. The backend is needed to populate the DB for the Crawler.

Keep in mind though, if you don't spin up a DAPS, the Crawler will start but you won't see it actually working since it needs to talk to other connectors (which means it also needs to talk to the DAPS).

Let me know if it worked out for you