Question: does DataProfiler support deeply nested arrays?

[tcCeligo]

First, let me say "this is an awesome project". It has sooo much of what i'm looking for.

Now, back to the question... is the code capable of drilling down into the structure of arrays? I can get something like this from PySpark's printSchema() command (using a sample schema )

jsonDF.printSchema()
root
|-- _corrupt_record: string (nullable = true)
|-- f1: string (nullable = true)
|-- f2: string (nullable = true)
|-- f3: struct (nullable = true)
| |-- address: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- line: string (nullable = true)
| | | |-- value: string (nullable = true)
| |-- city: string (nullable = true)
| |-- state_cd: string (nullable = true)
| |-- zip: string (nullable = true)

You can see the "f3.address.element.line" and "f3.address.element.value" are defined. In DataProfiler, it does not drill into the address array to define the line and value elements:

  {
        "column_name": "f3.address",
        "data_type": "string",
        "data_label": "UNKNOWN",
        "categorical": true,
        "order": "descending",
        "samples": "[\"[{'line': '1', 'value': '100 Main St'}, {'line': 2, 'value': 'Apt 123'}]\",\n \"[{'line': '1', 'value': '123 Park Rd'}, {'line': 2, 'value': 'Apt 3'}, {'line': 3, 'value': 'c/o Tony'}]\",\n \"[{'line': '1', 'value': '123 Park Rd'}, {'line': 2, 'value': 'Apt 3'}, {'line': 3, 'value': 'c/o Tony'}]\"]",
        "statistics": {
            "min": 72.0,
            "max": 104.0,
            "mode": "[103.984]",
            "median": 103.976,
            "sum": 280.0,
            "mean": 93.3333,
            "variance": 341.3333,
            "stddev": 18.4752,
            "skewness": -1.7321,
            "kurtosis": NaN,
            "quantiles": {
                "0": 72.024,
                "1": 103.976,
                "2": 103.988
            },
            "median_abs_deviation": 0.016,
            "vocab": "[',', 'o', '2', 'k', 'a', ... , '0', 'S', 'd', ':', 'P']",
            "unique_count": 2,
            "unique_ratio": 0.6667,
            "categories": "[\"[{'line': '1', 'value': '123 Park Rd'}, {'line': 2, 'value': 'Apt 3'}, {'line': 3, 'value': 'c/o Tony'}]\",\n \"[{'line': '1', 'value': '100 Main St'}, {'line': 2, 'value': 'Apt 123'}]\"]",
            "gini_impurity": 0.4444,
            "unalikeability": 0.6667,
            "categorical_count": {
                "[{'line': '1', 'value': '123 Park Rd'}, {'line': 2, 'value': 'Apt 3'}, {'line': 3, 'value': 'c/o Tony'}]": 2,
                "[{'line': '1', 'value': '100 Main St'}, {'line': 2, 'value': 'Apt 123'}]": 1
            },
            "sample_size": 110016,
            "null_count": 110013,
            "null_types": "['nan']",
            "data_type_representation": {
                "datetime": 0.0,
                "int": 0.0,
                "float": 0.0,
                "string": 1.0
            }
        }
    }

Am I missing any property that would allow it to drill further down into arrays like that?

[taylorfturner]

Thanks for your kinds words, @tcCeligo!

Thanks for your detailed issue you've raised here -- the use case you have laid out here for more nested data profiling it definitely interesting.

Suffice it to say, I don't think there is a super clean way to do this at the moment with the profiler.

Is it possible for you to flatten the jsonDF you referenced in your example? Such that you go from

jsonDF.printSchema()
root
|-- _corrupt_record: string (nullable = true)
|-- f1: string (nullable = true)
|-- f2: string (nullable = true)
|-- f3: struct (nullable = true)
| |-- address: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- line: string (nullable = true)
| | | |-- value: string (nullable = true)
| |-- city: string (nullable = true)
| |-- state_cd: string (nullable = true)
| |-- zip: string (nullable = true)

to

jsonDF.printSchema()
root
|-- _corrupt_record: string (nullable = true)
|-- f1: string (nullable = true)
|-- f2: string (nullable = true)
|-- f3: struct (nullable = true)
|-- f3_address_line: string (nullable = true)
|-- f3_address_value: string(nullable=true)
|-- f3_city: string (nullable = true)
|-- f3_state_cd: string (nullable = true)
|-- f3_zip: string (nullable = true)

We are more than open to proposals for new feature and new ideas as well @tcCeligo

[tcCeligo]

In our use case, we will have 100s of extracts that we will need to process. The requirements of that process will be to normalize the data that comes in through those extracts - which requires I can find and flatten the arrays. In another phase of the processing I will need to take action on the fields and that will vary based on the derived/inferred data types.

I can do an initial pass with pyspark to get the information required in order to normalize, and then a second pass over the data with DataProfiler to get the precision on the data type inference ... but who wants to do two passes over data :)

[taylorfturner]

Yeah, that's totally fair -- I think the short and sweet of it is that current state doesn't support the nested columns.

I would try to flatten the dataframe / columns, personally, @tcCeligo.

Thanks for reaching out and if you are interested in contributing support for nested columns, we'd be glad to entertain a new feature proposal. Thanks!