diff --git a/docs/index.md b/docs/index.md index 1c27b96d..b6c8a0de 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,9 +17,10 @@ We work on the following projects within the _Transparency of Algorithmic Decisi graph TB ak[Algoritmekader] <--> amt - subgraph amt[Algorithm Management Toolkit] + subgraph amt[Algorithm Management Toolkit] tr[Task Registry] --> amp[Algorithm Management Platform] st[Reporting Standard] --> amp + amp <--> ai_act_decision_tree[AI Act Decision Tree] amp <--> llm[LLM Benchmark Tooling] end diff --git a/docs/projects/amt/adrs/0008-systemcard-storage.md b/docs/projects/amt/adrs/0008-systemcard-storage.md new file mode 100644 index 00000000..1b28bced --- /dev/null +++ b/docs/projects/amt/adrs/0008-systemcard-storage.md @@ -0,0 +1,54 @@ +# AMT-0008 System card Storage + +## Context + +By default, Kubernetes pods use ephemeral storage, which is tied to the pod's lifecycle. +When the pod terminates or restarts, all data is lost. The `/tmp/` directory, being part of +the system's temporary file storage, is cleared during reboots or pod restarts, resulting in +the deletion of system_cards. Therefore, we need a different kind of storage to preserve the data. + +## Assumptions + +* The system card data is small to moderate in size (up to 255MB), making it manageable to store +in databases (in postgres as well as in in SQLite). +* Tracking changes to the system card data over time is not a priority in the short term, but may become +necessary in the future. + +## Decision + +The system card of an algorithm system is stored solely as a JSON blob in the projects table in Postgres, +with no additional storage elsewhere. + +## Risks + +* **Data Overwrite**: As the system card is overwritten with each update, it becomes difficult to track +historical changes or revert to previous states. +* **Scaling**: As the project grows, managing larger JSON blobs may present performance challenges, +particularly when handling complex queries. +* **Collaboration**: Collaborating on the system card content is more difficult, as the JSON format +requires parsing and manual intervention for certain tasks. +* **Limited Querying**: While Postgres supports querying and indexing JSON fields, complex queries and data manipulations +may be inefficient without proper indexing or further optimization. + +## Consequences + +### Positive + +* **Fast implementation**: The solution is easy to set up, reducing the time to get the project operational. +* **Future proof**: This approach is designed with future scalability in mind. While system cards will initially +be stored in Postgres as JSONB blobs, we anticipate migrating to a Git-based local or remote storage solution +as the system evolves. Importantly, this initial decision allows for a seamless transition in the future, +ensuring no obstacles to migration. +* **Single source & Fast access**: Centralizing everything in a single Postgres database streamlines backups, +reduces maintenance complexity, and ensures quick data access. +* **Built-in permissions**: Postgres provides built-in access control and security through its permission system. + +### Negative + +* **Data tracking**: Changes to the system card are overwritten, making it difficult to maintain a history or audit trail. +* **Complex queries**: Complex queries can be inefficient and require custom parsing. +* **Collaboration**: Collaborating on the JSONB data is challenging due to its complex format and lack of version control. +* **Scalability**: As the JSONB blobs grow in size, the storage overhead and query performance may become significant issues. +* **Not supported by SQLite**: While SQLite supports JSON through its JSON1 extension, it does not support PostgreSQL's +JSONB data type natively, which complicates local development and testing environments that rely on SQLite as a database +backend. diff --git a/docs/projects/amt/index.md b/docs/projects/amt/index.md index 8963cc04..4b2391a8 100644 --- a/docs/projects/amt/index.md +++ b/docs/projects/amt/index.md @@ -1,19 +1,20 @@ # AMT -AMT is the acronym for Algorithm Management Toolkit. AMT has the goal to make algorithmic -systems more transparent; it achieves this by generating standardized reports on the algorithmic system which -encompasses both technical aspects in addition to descriptive information about the system and regulatory assessments. -For both the system and the model the lifecycle is important and this needs to be taken into account. The definition -for an algorithm is derived from the [Algoritmeregister](https://algoritmes.overheid.nl/nl/footer/over-algoritmes). +AMT is the acronym for [Algorithm Management Toolkit](https://amt.prd.apps.digilab.network). +The AMT aims to enhance transparency and governance throughout +the entire lifecycle of algorithmic systems. By generating standardized reports, AMT provides a comprehensive view +of both technical details and descriptive information, including regulatory assessments, from development to deployment +and beyond. This continuous approach promotes accountability, oversight, and collaboration, ensuring that both models +and data remain transparent, controlled, and validated over time. The definition for an algorithm is derived from the +[Algoritmeregister](https://algoritmes.overheid.nl/nl/footer/over-algoritmes). -One of the goals of the TAD project is providing a standardized format of reporting on an algorithmic +One of the goals of the AMT is providing a standardized format of reporting on an algorithmic system by developing a [Reporting Standard](reporting-standard/index.md). This Reporting Standard consists out of a [System Card](reporting-standard/index.md#system_card) which contains [Model Cards](reporting-standard/index.md#model_card) and [Assessment Cards](reporting-standard/index.md#assessment_card). -The final result of the project is producing System, Model and Assessment Cards with both performance metrics -and technical measurements on fairness and bias of the model, assessments on the system where the specific -algorithm resides, and descriptive information about the system. +The final result of the AMT is producing System, Model and Assessment Cards with performance metrics, (regulatory) +assessments on the system where the specific algorithm resides, and descriptive information about the system. The requirements and instruments are dictated by the [Algoritmekader](https://minbzk.github.io/Algoritmekader/).