Add DataFlows, define egress and ingress on Systems and `Priv…

…acyDeclaration`s (#85) * Define the `DataFlow` resource model And the `FlowableResources` helper `Enum` * System resources may define `egress` and `ingress` * PrivacyDeclarations may define `egress`/`ingress` * Validate PrivacyDeclaration `egress`/`ingress` * Clean up the System model test, expose `DataFlow` Also fixes a circular import * Test System creation with `egress`/`ingress` * Update `CHANGELOG.md` * Update docs * Appease the almighty pylint * Clarify the changelog * Test that errors are raised for invalid Systems * Test that `egress` and `ingress` are optional * Improve `egress`/`ingress` descriptions * Shore up `PrivacyDeclaration` API docs * Deprecate `system_dependencies` This field is obsolete with the addition of `egress` and `ingress`. * Deprecate `dataset_references` This field is obsolete with the addition of `egress` and `ingress`. * Validate the "user" `DataFlow` special case * Test the user ingress case for `System`s * Add deprecations to `CHANGELOG.md`
IABTechLab · Sep 20, 2022 · f45fa28 · f45fa28
1 parent 40e7417
commit f45fa28
Show file tree

Hide file tree

Showing 14 changed files with 415 additions and 36 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,10 +16,25 @@ The types of changes are:
 
 ## [Unreleased](https://github.com/ethyca/fideslang/compare/1.1.0...main)
 
+### Added
+
+* The `DataFlow` resource model defines a resource with which a `System` resource may communicate [#85](https://github.com/ethyca/fideslang/pull/85)
+* `PrivacyDeclaration`s may define `egress` and `ingress`, to contextualize communications with other resources [#85](https://github.com/ethyca/fideslang/pull/85)
+
+### Deprecated
+
+* The `dataset_references` field of `PrivacyDeclaration` resources [#85](https://github.com/ethyca/fideslang/pull/85)
+* The `system_dependencies` field of `System` resources [#85](https://github.com/ethyca/fideslang/pull/85)
+
+### Developer Experience
+
+* The `DataFlow` resource model is exposed when importing `fideslang` [#85](https://github.com/ethyca/fideslang/pull/85)
+
 ### Fixed
 
 * Fixed broken links in docs [#74](https://github.com/ethyca/fideslang/pull/74)
 * Pydantic 1.10.0 was causing issues so specified the pydantic version needs to be less than 1.10.0 [#79](https://github.com/ethyca/fideslang/pull/79)
+* Resolved a circular import in `default_taxonomy.py` [#85](https://github.com/ethyca/fideslang/pull/85)
 
 ## [1.2.0](https://github.com/ethyca/fideslang/compare/1.1.0...1.2.0)
 

diff --git a/demo_resources/demo_system.yml b/demo_resources/demo_system.yml
@@ -12,6 +12,9 @@ system:
       is_required: True
       progress: Complete
       link: https://example.org/analytics_system_data_protection_impact_assessment
+    ingress:
+      - fides_key: demo_users_dataset
+        type: dataset
     privacy_declarations:
       - name: Analyze customer behaviour for improvements.
         data_categories:
@@ -21,7 +24,7 @@ system:
         data_subjects:
           - customer
         data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
-        dataset_references:
+        ingress:
           - demo_users_dataset
 
   - fides_key: demo_marketing_system

diff --git a/mkdocs/docs/resources/system.md b/mkdocs/docs/resources/system.md
@@ -54,6 +54,14 @@ The array of properties that declare the requirement for and information surroun
 
 Information will be exported as part of the data map or Record of Processing Activites (RoPA)
 
+**egress**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[array]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+
+The resources to which the System sends data.
+
+**ingress**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[array]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+
+The resources from which the System receives data.
+
 **privacy_declarations**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[array]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 
 The array of declarations describing the types of data in your system. This is a list of the privcy attributes (`data_category`, `data_use`, `data_subject`, and `data_qualifier`) for each of your systems.
@@ -88,6 +96,16 @@ system:
       is_required: True
       progress: Complete
       link: https://example.org/analytics_system_data_protection_impact_assessment
+    egress:
+      - fides_key: another_demo_system
+        type: system
+        data_categories:
+          - user.contact
+    ingress:
+      - fides_key: yet_another_demo_system
+        type: system
+        data_categories:
+          - user.device.cookie_id
     privacy_declarations:
       - name: Analyze customer behaviour for improvements.
         data_categories:
@@ -97,8 +115,10 @@ system:
         data_subjects:
           - customer
         data_qualifier: identified_data
-        dataset_references:
-          - demo_users_dataset
+        egress:
+          - another_demo_system
+        ingress:
+          - yet_another_demo_system
 ```
 
 **Demo manifest file:** `/fides/fidesctl/demo_resources/demo_system.yml`
@@ -121,6 +141,20 @@ system:
     "email": "[email protected]",
     "phone": "+1 555 555 5555"
   },
+  "egress": [
+    {
+      "fides_key": "another_demo_system",
+      "type": "system",
+      "data_categories": ["user.contact"]
+    }
+  ],
+  "ingress": [
+    {
+      "fides_key": "yet_another_demo_system",
+      "type": "system",
+      "data_categories": ["user.device.cookie_id"]
+    }
+  ],
   "privacy_declarations": [
     {
       "name": "Analyze customer behaviour for improvements.",
@@ -133,9 +167,8 @@ system:
         "customer"
       ],
       "data_qualifier": "identified_data",
-      "dataset_references": [
-        "demo_users_dataset"
-      ]
+      "egress": ["another_demo_system"],
+      "ingress": ["yet_another_demo_system"]
     }
   ]
 }

diff --git a/src/fideslang/__init__.py b/src/fideslang/__init__.py
@@ -4,9 +4,13 @@
 
 from typing import Dict, Type, Union
 
+from .default_fixtures import COUNTRY_CODES
+from .default_taxonomy import DEFAULT_TAXONOMY
+
 # Export the Models
 from .models import (
     DataCategory,
+    DataFlow,
     DataQualifier,
     Dataset,
     DatasetField,
@@ -25,9 +29,6 @@
     Taxonomy,
 )
 
-from .default_fixtures import COUNTRY_CODES
-from .default_taxonomy import DEFAULT_TAXONOMY
-
 FidesModelType = Union[Type[FidesModel], Type[Evaluation]]
 model_map: Dict[str, FidesModelType] = {
     "data_category": DataCategory,

diff --git a/src/fideslang/default_taxonomy.py b/src/fideslang/default_taxonomy.py
@@ -1,6 +1,6 @@
 """This module contains the the default resources that Fideslang ships with."""
 
-from fideslang import (
+from .models import (
     DataCategory,
     DataQualifier,
     DataSubject,

diff --git a/src/fideslang/models.py b/src/fideslang/models.py
@@ -5,6 +5,7 @@
 
 from enum import Enum
 from typing import Dict, List, Optional
+from warnings import warn
 
 from pydantic import AnyUrl, BaseModel, Field, HttpUrl, root_validator, validator
 
@@ -645,6 +646,27 @@ class PrivacyDeclaration(BaseModel):
     dataset_references: Optional[List[FidesKey]] = Field(
         description="Referenced Dataset fides keys used by the system.",
     )
+    egress: Optional[List[FidesKey]] = Field(
+        description="The resources to which data is sent. Any `fides_key`s included in this list reference `DataFlow` entries in the `egress` array of any `System` resources to which this `PrivacyDeclaration` is applied."
+    )
+    ingress: Optional[List[FidesKey]] = Field(
+        description="The resources from which data is received. Any `fides_key`s included in this list reference `DataFlow` entries in the `ingress` array of any `System` resources to which this `PrivacyDeclaration` is applied."
+    )
+
+    @validator("dataset_references")
+    @classmethod
+    def deprecate_dataset_references(cls, value: List[FidesKey]) -> List[FidesKey]:
+        """
+        Warn that the `dataset_references` field is deprecated, if set.
+        """
+
+        if value is not None:
+            warn(
+                "The dataset_references field is deprecated, and will be removed in a future version of fideslang. Use the 'egress' and 'ingress` fields instead.",
+                DeprecationWarning,
+            )
+
+        return value
 
 
 class SystemMetadata(BaseModel):
@@ -665,6 +687,51 @@ class SystemMetadata(BaseModel):
     )
 
 
+class FlowableResources(str, Enum):
+    """
+    The resource types with which DataFlows can be created.
+    """
+
+    DATASET = "dataset"
+    SYSTEM = "system"
+    USER = "user"
+
+
+class DataFlow(BaseModel):
+    """
+    The DataFlow resource model.
+
+    Describes a resource model with which a given System resource communicates.
+    """
+
+    fides_key: FidesKey = Field(
+        ...,
+        description="Identifies the System or Dataset resource with which the communication occurs. May also be 'user', to represent communication with the user(s) of a System.",
+    )
+    type: FlowableResources = Field(
+        ...,
+        description="Specifies the resource model class for which the `fides_key` applies. May be any of 'dataset', 'system', or 'user'.",
+    )
+    data_categories: Optional[List[FidesKey]] = Field(
+        description="An array of data categories describing the data in transit.",
+    )
+
+    @root_validator()
+    @classmethod
+    def user_special_case(cls, values: Dict) -> Dict:
+        """
+        If either the `fides_key` or the `type` are set to "user",
+        then the other must also be set to "user".
+        """
+
+        if values["fides_key"] == "user" or values["type"] == "user":
+            assert (
+                values["fides_key"] == "user" and values["type"] == "user"
+            ), "The 'user' fides_key is required for, and requires, the type 'user'"
+
+        return values
+
+
 class System(FidesModel):
     """
     The System resource model.
@@ -688,6 +755,12 @@ class System(FidesModel):
         default=DataResponsibilityTitle.CONTROLLER,
         description=DataResponsibilityTitle.__doc__,
     )
+    egress: Optional[List[DataFlow]] = Field(
+        description="The resources to which the System sends data."
+    )
+    ingress: Optional[List[DataFlow]] = Field(
+        description="The resources from which the System receives data."
+    )
     privacy_declarations: List[PrivacyDeclaration] = Field(
         description=PrivacyDeclaration.__doc__,
     )
@@ -719,6 +792,49 @@ class System(FidesModel):
 
     _check_valid_country_code: classmethod = country_code_validator
 
+    @validator("privacy_declarations", each_item=True)
+    @classmethod
+    def privacy_declarations_reference_data_flows(
+        cls,
+        value: PrivacyDeclaration,
+        values: Dict,
+    ) -> PrivacyDeclaration:
+        """
+        Any `PrivacyDeclaration`s which include `egress` and/or `ingress` fields must
+        only reference the `fides_key`s of defined `DataFlow`s in said field(s).
+        """
+
+        for direction in ["egress", "ingress"]:
+            fides_keys = getattr(value, direction, None)
+            if fides_keys is not None:
+                data_flows = values[direction]
+                system = values["fides_key"]
+                assert (
+                    data_flows is not None and len(data_flows) > 0
+                ), f"PrivacyDeclaration '{value.name}' defines {direction} with one or more resources and is applied to the System '{system}', which does not itself define any {direction}."
+
+                for fides_key in fides_keys:
+                    assert fides_key in [
+                        data_flow.fides_key for data_flow in data_flows
+                    ], f"PrivacyDeclaration '{value.name}' defines {direction} with '{fides_key}' and is applied to the System '{system}', which does not itself define {direction} with that resource."
+
+        return value
+
+    @validator("system_dependencies")
+    @classmethod
+    def deprecate_system_dependencies(cls, value: List[FidesKey]) -> List[FidesKey]:
+        """
+        Warn that the `system_dependencies` field is deprecated, if set.
+        """
+
+        if value is not None:
+            warn(
+                "The system_dependencies field is deprecated, and will be removed in a future version of fideslang. Use the 'egress' and 'ingress` fields instead.",
+                DeprecationWarning,
+            )
+
+        return value
+
     class Config:
         "Class for the System config"
         use_enum_values = True

diff --git a/tests/conftest.py b/tests/conftest.py
@@ -118,10 +118,8 @@ def resources_dict():
                     data_use="provide",
                     data_subjects=[],
                     data_qualifier="aggregated_data",
-                    dataset_references=[],
                 )
             ],
-            system_dependencies=[],
         ),
     }
     yield resources_dict

diff --git a/tests/data/failing_dataset_collection_taxonomy.yml b/tests/data/failing_dataset_collection_taxonomy.yml
@@ -20,6 +20,9 @@ system:
     name: Customer Data Sharing System
     description: Share data about our users with third-parties for advertising
     system_type: Service
+    ingress:
+      - fides_key: test_db_dataset_failing_dataset
+        type: dataset
     privacy_declarations:
       - name: Share Political Opinions
         data_categories:
@@ -28,9 +31,8 @@ system:
         data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
         data_subjects:
           - customer
-        dataset_references:
+        ingress:
           - test_db_dataset_failing_dataset
-    system_dependencies: []
 
 policy:
   - fides_key: primary_privacy_policy

diff --git a/tests/data/failing_dataset_field_taxonomy.yml b/tests/data/failing_dataset_field_taxonomy.yml
@@ -21,6 +21,9 @@ system:
     name: Customer Data Sharing System
     description: Share data about our users with third-parties for advertising
     system_type: Service
+    ingress:
+      - fides_key: test_db_dataset_failing_dataset
+        type: dataset
     privacy_declarations:
       - name: Share Political Opinions
         data_categories:
@@ -29,9 +32,8 @@ system:
         data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
         data_subjects:
           - customer
-        dataset_references:
+        ingress:
           - test_db_dataset_failing_dataset
-    system_dependencies: []
 
 policy:
   - fides_key: primary_privacy_policy

diff --git a/tests/data/failing_dataset_taxonomy.yml b/tests/data/failing_dataset_taxonomy.yml
@@ -20,6 +20,9 @@ system:
     name: Customer Data Sharing System
     description: Share data about our users with third-parties for advertising
     system_type: Service
+    ingress:
+      - fides_key: test_db_dataset_failing_dataset
+        type: dataset
     privacy_declarations:
       - name: Share Political Opinions
         data_categories:
@@ -28,9 +31,8 @@ system:
         data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
         data_subjects:
           - customer
-        dataset_references:
+        ingress:
           - test_db_dataset_failing_dataset
-    system_dependencies: []
 
 policy:
   - fides_key: primary_privacy_policy

diff --git a/tests/data/failing_declaration_taxonomy.yml b/tests/data/failing_declaration_taxonomy.yml
@@ -11,7 +11,6 @@ system:
         data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
         data_subjects:
           - customer
-    system_dependencies: []
 
 policy:
   - fides_key: primary_privacy_policy