canton v2.8.8
Release of Canton 2.8.8
Canton 2.8.8 has been released on June 26, 2024. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory.
Please also consult the full documentation of this release.
Summary
This is a maintenance release including bug fixes: two in the repair service and one memory leak with metrics collection.
Bugfixes
(24-012, Major): repair.purge appears not to clean up key indices in cache
Issue Description
When contracts containing keys are purged using the repair service, the corresponding contract keys remain in the ledger API cache. Then, when a subsequent client command requests to lookup the contract behind the spent key, it is rejected already at the interpretation time in the originating participant
Affected Deployments
Participant
Affected Versions
All 2.3-2.7, 2.8.0-2.8.7
Impact
Participant is unable to reuse the affected keys in subsequent commands and requires a restart
Symptom
Participant log file contains the following error when new transaction tries to use the spent key: NOT_FOUND: CONTRACT_NOT_FOUND(11,a61f3d3c): Contract could not be found with id...
Workaround
Restart the participant as the problem only affects the in-memory data
Likeliness
Repair purge must be used followed by an attempt to use the key in a follow-up transaction
Recommendation
Users planning to use repair service should upgrade to 2.8.8
(24-013, Minor): repair.migrate_domain prevents participant pruning
Issue Description
After a hard domain migration the source domain RequestJournalStore clean head check throws an IllegalArgumentException on behalf of the inactive domain and prevents participant pruning.
Affected Deployments
Participant
Affected Versions
All 2.3-2.7, 2.8.0-2.8.7
Impact
Participant is unable to prune.
Symptom
Participant log file contains the following errors when attempting to prune: IllegalArgumentException: Attempted to prune at timestamp which is not earlier than _ associated with the clean head
Workaround
None except manually updating the RequestJournalStore database to set an artificially large clean head counter
Likeliness
Likely after a hard domain migration
Recommendation
Users are advised to upgrade to 2.8.8
(24-014, Major): Memory leak in the metrics associated with the grpc statistics
Issue Description
Canton components slowly accumulate memory that is not reclaimed in garbage collection. Typically it is at a rate of 250MB per week. This happens only if prometheus exporter is configured in the canton configuration file and a periodic metric read is performed by a prometheus agent.
Affected Deployments
All node types
Affected Versions
All 2.3-2.7, 2.8.0-2.8.7
Impact
Canton node can crash with an out-of-memory error (OOM).
Symptom
Node crashes. Memory dump shows an excessive memory consumption for io.opentelemetry.api.internal.InternalAttributeKeyImpl.
Workaround
Run a health dump on an affected node. This drains the collected metrics and reclaims associated memory.
Likeliness
Likely for anyone using prometheus agent.
Recommendation
Users are advised to upgrade to 2.8.8
Compatibility
The following Canton protocol versions are supported:
| Dependency | Version |
|---|---|
| Canton protocol versions | 3, 4, 5 |
Canton has been tested against the following versions of its dependencies:
| Dependency | Version |
|---|---|
| Java Runtime | OpenJDK 64-Bit Server VM Zulu11.70+15-CA (build 11.0.22+7-LTS, mixed mode) |
| Postgres | Recommended: PostgreSQL 12.19 (Debian 12.19-1.pgdg120+1) – Also tested: PostgreSQL 11.16 (Debian 11.16-1.pgdg90+1), PostgreSQL 13.15 (Debian 13.15-1.pgdg120+1), PostgreSQL 14.12 (Debian 14.12-1.pgdg120+1), PostgreSQL 15.7 (Debian 15.7-1.pgdg120+1) |
| Oracle | 19.20.0 |