Merge branch 'object-v0' of https://github.com/MTES-MCT/wikicarbone i…

…nto object-v0

.github/pull_request_template.md

@@ -0,0 +1,16 @@
+## :wrench: Problem
+
+> _Describe the problem you're trying to solve, the reason why you are creating this pull request. Provide a link to the Notion card if any._
+
+## :cake: Solution
+
+> _Explain the solution that this PR implements to solve the problem above._
+
+
+## :rotating_light:  Points to watch/comments
+
+> _If there is anything unusual or some clarifications needed for the reviewer to better understand the PR, discuss it here._
+
+## :desert_island: How to test
+
+> _What someone else than you should do to validate that the solution you implemented is working as expected. Don't hesitate to be too verbose and to explain in details, using bullet points for example, the steps to follow to test the expected behavior._

.github/workflows/node.js.yml

@@ -53,9 +53,7 @@ jobs:
       uses: actions/cache@v4
       with:
         path: ~/.local/share/virtualenvs
-        key: ${{ runner.os }}-${{ matrix.python-version }}-pipenv-${{ hashFiles('Pipfile.lock') }}
-        restore-keys: |
-          ${{ runner.os }}-pipenv-
+        key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-pipenv-${{ hashFiles('Pipfile.lock') }}
 
     - name: Install Node dependencies
       run: npm ci --prefer-offline --no-audit

CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## [2.3.0](https://github.com/MTES-MCT/ecobalyse/compare/v2.2.0...v2.3.0) (2024-09-25)
+
+
+### Features
+
+* add link to changelog in app footer. ([#748](https://github.com/MTES-MCT/ecobalyse/issues/748)) ([efe88f5](https://github.com/MTES-MCT/ecobalyse/commit/efe88f57d4f61e74e84f62c5334a89d36fd767ee))
+* airTransportRatio should depend on durability ([#757](https://github.com/MTES-MCT/ecobalyse/issues/757)) ([a0761d1](https://github.com/MTES-MCT/ecobalyse/commit/a0761d169469fddff7b9158f21de26d96a98ae0f))
+* displayName in the textile explorer, reordered columns ([#737](https://github.com/MTES-MCT/ecobalyse/issues/737)) ([65d0ed5](https://github.com/MTES-MCT/ecobalyse/commit/65d0ed547566c193c5617147a35604e26b0bbe6d))
+
+
+### Bug Fixes
+
+* **api:** handle ingredient plane transport in food POST api. ([#769](https://github.com/MTES-MCT/ecobalyse/issues/769)) ([62587e2](https://github.com/MTES-MCT/ecobalyse/commit/62587e23593e459d66726b2221932092393790e5))
+* check db integrity after building it ([#753](https://github.com/MTES-MCT/ecobalyse/issues/753)) ([5b41ef6](https://github.com/MTES-MCT/ecobalyse/commit/5b41ef6a6e1d4e799ed2abe7f21a321a87f9ae83))
+* check uniqueness of JSON db primary keys at build time. ([#766](https://github.com/MTES-MCT/ecobalyse/issues/766)) ([0927954](https://github.com/MTES-MCT/ecobalyse/commit/0927954dfe472557f0e7c2e5e4aa24d1ce8572c2))
+* decode and validate all optionals. ([#764](https://github.com/MTES-MCT/ecobalyse/issues/764)) ([87a7c6a](https://github.com/MTES-MCT/ecobalyse/commit/87a7c6af3e6edc12c2daa36192ad7f18fdefc444))
+* encode physicalDurability parameter. ([#751](https://github.com/MTES-MCT/ecobalyse/issues/751)) ([f6750b8](https://github.com/MTES-MCT/ecobalyse/commit/f6750b8aea6dc0a4500a23465bfdbc0f0b627743))
+* fix github CI python build setup. ([#762](https://github.com/MTES-MCT/ecobalyse/issues/762)) ([ea2cd9f](https://github.com/MTES-MCT/ecobalyse/commit/ea2cd9ff566129081ccf37caef25377717933c9d))
+* fixed brightway explorer notebook error (wrong key) ([#745](https://github.com/MTES-MCT/ecobalyse/issues/745)) ([bc436c2](https://github.com/MTES-MCT/ecobalyse/commit/bc436c2d1520efb9de269cdadccf6587d1904468))
+* in brightway explorer: improve display of compartment categories, if any ([#754](https://github.com/MTES-MCT/ecobalyse/issues/754)) ([757d5a6](https://github.com/MTES-MCT/ecobalyse/commit/757d5a6b50363995011e23bd5719adedb44d296f))
+* stricter validation of POST json body passed to the textile API. ([#760](https://github.com/MTES-MCT/ecobalyse/issues/760)) ([a85bd8a](https://github.com/MTES-MCT/ecobalyse/commit/a85bd8aa506ce87b9b1310b6a4a933a591ce442e))
+* **textile:** distribution step had no inland road transports added. ([#761](https://github.com/MTES-MCT/ecobalyse/issues/761)) ([d789d7d](https://github.com/MTES-MCT/ecobalyse/commit/d789d7d63a3dede6a4c2b07d43f6f43b9328a519))
+* Update export outside of EU probability. ([#765](https://github.com/MTES-MCT/ecobalyse/issues/765)) ([c3fd9f2](https://github.com/MTES-MCT/ecobalyse/commit/c3fd9f2d5d0cc01232b31f9aa1ef657b33796292))
+
 ## [2.2.0](https://github.com/MTES-MCT/ecobalyse/compare/v2.1.1...v2.2.0) (2024-09-12)
 
 

bin/build-db

@@ -22,35 +22,60 @@ try {
   process.exit(1);
 }
 
-function getJson(path) {
+function parseAndValidate(path, idKeyName = "id") {
   const raw = JSON.parse(fs.readFileSync(path).toString());
+  if (idKeyName && Array.isArray(raw)) {
+    try {
+      validatePrimaryKeys(raw, idKeyName);
+    } catch (err) {
+      console.error(`🚨 ERROR building ${path}:\n   ${err}`);
+      process.exit(1);
+    }
+  }
   // Adapts a standard JSON string to what is expected to be the format
   // used in Elm's template strings (`"""{}"""`).
   return JSON.stringify(raw).replaceAll("\\", "\\\\");
 }
 
+/**
+ * Validates that unique identifiers are actually unique in provided datasource.
+ */
+function validatePrimaryKeys(records, idKeyName) {
+  const ids = records.map((record) => record[idKeyName]).sort();
+  const duplicates = ids.filter((item, index) => ids.indexOf(item) !== index);
+  if (duplicates.length > 0) {
+    throw new Error(`Duplicate ${idKeyName}: ${duplicates}`);
+  }
+}
+
 const targetDbFile = "src/Static/Json.elm";
 const elmTemplate = fs.readFileSync(`${targetDbFile}-template`).toString();
 const elmWithFixtures = elmTemplate
   // Transverse JSON data
-  .replace("%countriesJson%", getJson("public/data/countries.json"))
-  .replace("%impactsJson%", getJson("public/data/impacts.json"))
-  .replace("%transportsJson%", getJson("public/data/transports.json"))
+  .replace("%countriesJson%", parseAndValidate("public/data/countries.json", "code"))
+  .replace("%impactsJson%", parseAndValidate("public/data/impacts.json"))
+  .replace("%transportsJson%", parseAndValidate("public/data/transports.json"))
   // Food JSON data
-  .replace("%foodIngredientsJson%", getJson("public/data/food/ingredients.json"))
+  .replace("%foodIngredientsJson%", parseAndValidate("public/data/food/ingredients.json", "id"))
   .replace(
     "%foodProcessesJson%",
-    getJson(NODE_ENV === "test" ? dataFiles.foodDetailed : dataFiles.foodNoDetails),
+    parseAndValidate(NODE_ENV === "test" ? dataFiles.foodDetailed : dataFiles.foodNoDetails, "id"),
   )
-  .replace("%foodProductExamplesJson%", getJson("public/data/food/examples.json"))
+  .replace("%foodProductExamplesJson%", parseAndValidate("public/data/food/examples.json", "id"))
   // Textile JSON data
-  .replace("%textileMaterialsJson%", getJson("public/data/textile/materials.json"))
+  .replace("%textileMaterialsJson%", parseAndValidate("public/data/textile/materials.json", "id"))
   .replace(
     "%textileProcessesJson%",
-    getJson(NODE_ENV === "test" ? dataFiles.textileDetailed : dataFiles.textileNoDetails),
+    parseAndValidate(
+      NODE_ENV === "test" ? dataFiles.textileDetailed : dataFiles.textileNoDetails,
+      "uuid",
+    ),
+  )
+  .replace(
+    "%textileProductExamplesJson%",
+    parseAndValidate("public/data/textile/examples.json", "id"),
   )
-  .replace("%textileProductExamplesJson%", getJson("public/data/textile/examples.json"))
-  .replace("%textileProductsJson%", getJson("public/data/textile/products.json"));
+  .replace("%textileProductsJson%", parseAndValidate("public/data/textile/products.json", "id"));
 
 const header =
   "---- THIS FILE WAS GENERATED FROM THE FILE `Json.elm-template` BY THE `/bin/build-db` SCRIPT";

data/Makefile

@@ -47,7 +47,7 @@ compare_food:
 	@$(call DOCKER,python3 export.py compare)
 
 format:
-	npm run format:json
+	npm run fix:all
 
 python:
 	echo Running Python inside the container...
@@ -94,4 +94,3 @@ clean_image:
 	docker image rm $(NAME)
 
 clean: clean_data clean_image
-

data/docker/entrypoint.sh

@@ -8,6 +8,11 @@ if [ $ECOBALYSE_ID -ne $JOVYAN_ID ]; then
     usermod -u $ECOBALYSE_ID jovyan
 fi
 
-chown -R 1000:100 "/home/jovyan/.npm"
+# Ensure .npm directory is owned by jovyan
+mkdir -p /home/jovyan/.npm
+chown -R jovyan:100 "/home/jovyan/.npm"
 
-gosu jovyan "$@"
+# Clear npm cache
+su jovyan -c "npm cache clean --force"
+
+exec gosu jovyan "$@"

data/notebooks/explore.py

@@ -112,12 +112,14 @@ def w_csv_button(contents, columns):
 def display_results(database, search, limit):
     """display the list of search results in the w_results widget"""
     results = list(bw2data.Database(database).search(search, limit=limit))
+    for a in results:
+        a["categories"] = ", ".join(a.get("categories", []))
     w_results.clear_output()
     w_details.clear_output()
     w_activity.options = [("", "")] + [
         (
-            str(i)
-            + f" {a.get('name', '')} {'(' if a.get('categories') else ''}{', '.join(a.get('categories', []))}{')' if a.get('categories') else ''}",
+            str(i) + f" {a.get('name', '')} "
+            f"{('(in ' + a.get('categories', []) + ')') if a.get('categories') else ''}",
             a,
         )
         for i, a in enumerate(results)
@@ -128,7 +130,7 @@ def display_results(database, search, limit):
         display(
             Markdown(f"## {('+' if len(results)==LIMIT else '')}{len(results)} results")
         )
-        columns = ["name", "code", "location"]
+        columns = ["name", "categories", "code", "location"]
         html = pandas.io.formats.style.Styler(
             pandas.DataFrame(results, columns=columns)
         )
@@ -651,7 +653,7 @@ def display_main_data(method, impact_category, activity):
             f"<li><b>Name</b>: {input_.get('name', 'N/A')}</li>"
             f"<li><b>Code</b>: {input_.get('code', 'N/A')}</li>"
             f"<li><b>Type</b>: {input_.get('type', 'N/A')}</li>"
-            f"<li><b>Categories</b>: {', '.join(input_.get('categories', 'N/A'))}</li>"
+            f"<li><b>Categories</b>: {', '.join(input_.get('categories', []))}</li>"
             f"<li><b>CAS number</b>: <a href=\"https://pubchem.ncbi.nlm.nih.gov/#query={str(input_.get('CAS number')).lstrip('0')}\">{str(input_.get('CAS number'))}</a></li>"
             f"<li><b>Unit</b>: {input_.get('unit', 'N/A')}</li>"
             f"<li><b>Id</b>: {input_.get('id', 'N/A')}</li>"
@@ -720,7 +722,8 @@ def display_main_data(method, impact_category, activity):
     w_details.clear_output()
     display(
         Markdown(
-            f"# 1 {activity.get('unit', '')} of {activity.get('name', '')} ({', '.join(activity.get('categories', 'N/A'))})"
+            f"# 1 {activity.get('unit', '')} of {activity.get('name', '')} "
+            f"{('(in ' + ', '.join(activity.get('categories', [])) + ')') if activity.get('categories') else ''}"
         )
     )
     display(

package.json

@@ -1,6 +1,6 @@
 {
   "name": "ecobalyse",
-  "version": "2.2.0",
+  "version": "2.3.0",
   "description": "Accélérer l'affichage environnemental de la filière textile française",
   "author": "Ecobalyse <[email protected]>",
   "license": "MIT",