Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Row-level filtering marking the records as pass when null values are present in the column #565

Open
eapframework opened this issue Apr 24, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@eapframework
Copy link

eapframework commented Apr 24, 2024

I am working on filtering data based on row-level checks. It's working fine when notnull values are present in the column

But incorrectly marking the records as pass when null values are present in the column.

For example

 import sparkSession.implicits._

    Seq(
      (1, "a", 1),
      (2, "b", 3),
      (3, null, null),
      (4, "c", 5),
      (5, null, null),
      (6, "d", 7)
    ).toDF("item", "att1", "att2")

Applied below rules:

rule1 : .isPrimaryKey("att1","att2")
rule2: .isGreaterThan("att2", "att1")
rule3: .isgreaterthanorequalto("att2", "att1")
+----+----+----+-----+-----+-----+
|item|att1|att2|rule1|rule2|rule3|
+----+----+----+-----+-----+-----+
|   1|   a|   1| true|false| true|
|   2|   b|   3| true| true| true|
|   3|null|null| true| true| true|
|   4|   c|   5| true| true| true|
|   5|null|null| true| true| true|
|   6|   d|   7| true| true| true|
+----+----+----+-----+-----+-----+

When columns values are null, the row-level check status is considered as true but it should be false.

@eapframework eapframework added the bug Something isn't working label Apr 24, 2024
@eapframework eapframework changed the title [BUG] [BUG] Row-level filtering marking the records as pass when null values are present in the column Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant