feat(pacer.email): improve bankruptcy short description parsing #1276

grossir · 2024-12-17T05:15:56Z

Solves #912, Solves #914

simplify parsing by getting rid of cases by court groups
support multi docket NEF parsing: add examples for deb, ctb, mdb, ndb, nhb, paeb, txnb
support flsb
correct wrong parsing for vaeb and mdb after double checking on PACER
updated paeb_1 example file where creation of example file had broken parsing

Solves #912, Solves #914 - simplify parsing by getting rid of cases by court groups - support multi docket NEF parsing: add examples for deb, ctb, mdb, ndb, nhb, paeb, txnb - support flsb - correct wrong parsing for vaeb and mdb after double checking on PACER - updated paeb_1 example file where creation of example file had broken parsing

mlissner

Wow, you really simplified the code while adding a bunch more test cases. Love it.

How much have you checked vs. the pacer history reports to make sure that your output is correct?

grossir

I left a screenshot of the "History/Documents" report on most JSON files to explain the changes. I noticed I need to improve the parsing of a couple of them

grossir · 2024-12-17T21:58:56Z

tests/examples/pacer/nef/s3/dcb_2.json

@@ -16,7 +16,7 @@
          "pacer_doc_id": null,
          "pacer_magic_num": null,
          "pacer_seq_no": null,
-          "short_description": "Hearing Held"
+          "short_description": "Hearing Held (BK)"


Document number 56 has short description "Hearing Held (BK)"

grossir · 2024-12-17T22:08:22Z

juriscraper/pacer/email.py

+
+        # Deletes:
+        # - extra docket number 'components', such as `federal_dn_judge_initials_assigned`
+        # - Chapter component


On deleting "Chapter", see nhb1 example
The email's subject is
Subject:21-10245-BAH Chapter 13 Mykle Lepene Affidavit of Compliance with Discharge Requirements
if we didn't delete the "Chapter" the short_description would be
"Chapter 13 Affidavit of Compliance with Discharge Requirements"
but it's different on pacer

grossir · 2024-12-17T22:09:29Z

tests/examples/pacer/nef/s3/paeb_2.json

@@ -16,7 +16,7 @@
          "pacer_doc_id": null,
          "pacer_magic_num": null,
          "pacer_seq_no": null,
-          "short_description": "CHAP - Hearing Continued"
+          "short_description": "CHAP - Hearing Continued (Bk Other)"


On keeping "CHAP" and "(Bk Other)"

grossir · 2024-12-17T22:10:14Z

tests/examples/pacer/nef/s3/vaeb_1.json

@@ -16,7 +16,7 @@
          "pacer_doc_id": "188040985133",
          "pacer_magic_num": "49963627",
          "pacer_seq_no": "101",
-          "short_description": "Notice of Dismissal"
+          "short_description": "Notice of Dismissal - CerDocTyp"


On keeping "CerDocTyp"

grossir · 2024-12-17T22:12:15Z

tests/examples/pacer/nef/s3/tnmb_1.json

@@ -16,7 +16,7 @@
          "pacer_doc_id": null,
          "pacer_magic_num": null,
          "pacer_seq_no": null,
-          "short_description": "Docket Order - Continue Hearing (Auto) Ch 13"
+          "short_description": "Docket Order - Continue Hearing (Auto)"


On deleting "Ch 13"

grossir · 2024-12-17T22:14:50Z

tests/examples/pacer/nef/s3/mdb_multi_2.json

+          "pacer_doc_id": null,
+          "pacer_magic_num": null,
+          "pacer_seq_no": null,
+          "short_description": "Close Adversary Case"


Subject in the email is
"Multiple Cases "Close Adversary Case" - AP -"

Should be reduced to "Close Adversary Case". It doesn't follow the usual clean up process, that's why I am using the early return

grossir · 2024-12-17T22:22:07Z

tests/examples/pacer/nef/s3/flsb_1.json

+          "pacer_doc_id": "050057572723",
+          "pacer_magic_num": "58443666",
+          "pacer_seq_no": "384",
+          "short_description": "UST Form 11"


This one should be "UST Form 11-MOR"

grossir · 2024-12-17T22:25:25Z

tests/examples/pacer/nef/s3/mdb_2.json

@@ -16,7 +16,7 @@
          "pacer_doc_id": "092051714783",
          "pacer_magic_num": "11582823",
          "pacer_seq_no": "409",
-          "short_description": "Order on Motion To Quash"
+          "short_description": "Order on Motion To Quash - CH"


This one shouldn't have the lagging " - CH"

grossir · 2024-12-17T22:26:15Z

tests/examples/pacer/nef/s3/mdb_multi_1.json

+          "pacer_doc_id": null,
+          "pacer_magic_num": null,
+          "pacer_seq_no": null,
+          "short_description": "Close Adversary Case"


This one is OK

grossir · 2024-12-17T22:27:44Z

tests/examples/pacer/nef/s3/paeb_1.json

@@ -4,7 +4,7 @@
  "court_id": "paeb",
  "dockets": [
    {
-      "case_name": "Br Cun",
+      "case_name": "Brittany Cunningham",


Editing the example source had messed up the parsing of this case

mlissner · 2024-12-18T18:00:09Z

Great. Thanks Gianfranco. Assigning to Alberto for review.

grossir · 2024-12-18T18:01:40Z

I am missing one more commit; I have been checking the old example files too and doing some minor updates

Keep "Chapter ..." string in paeb and pamb courts

for more information, see https://pre-commit.ci

grossir · 2024-12-18T20:12:38Z

tests/examples/pacer/nef/s3/pamb_4.json

+          "pacer_doc_id": "154018316665",
+          "pacer_magic_num": "88545762",
+          "pacer_seq_no": "31",
+          "short_description": "BNC Certificate of Notice (341 Meeting Notice (Chapter 13))"


We keep the "Chapter ..." string

Email subject:
Subject:Ch-13 1:24-bk-00941-HWV -Charlene R. House BNC Certificate of Notice (341 Meeting Notice (Chapter 13))

Document history report:

grossir · 2024-12-18T20:14:44Z

tests/examples/pacer/nef/s3/paeb_3.json

+          "pacer_doc_id": "152032523670",
+          "pacer_magic_num": "29282863",
+          "pacer_seq_no": "78",
+          "short_description": "Chapter 13 Plan"


We keep the "Chapter..." string

Email subject:
Subject:Ch-13 23-13878-pmm Chapter 13 Plan - Kervince Markenzy

History report

grossir · 2024-12-18T20:17:13Z

@albertisfu this now ready for review

albertisfu

Thanks @grossir I reviewed the refactor applied, and it looks good to me.
I just left a comment and a couple of concerns after finding some examples where the conditions for deciding when to keep or remove some suffixes and Chapter could be problematic.

albertisfu · 2024-12-24T01:00:15Z

juriscraper/pacer/email.py

-            return ""
+        # Some courts have subjects like
+        # `Multiple Cases "{docket} {case name} Close Adversary Case"`
+        if "close adversary case" in subject.lower():


I agree with this assumption. I looked for short descriptions that contain Close Adversary Case, and it seems safe to return just Close Adversary Case in these cases. However, I think we should also consider its quoted version, "Close Adversary Case", as seen in txnb_multi_1.json. The short description should simply be Close adversary case, as I confirmed in its docket history report.

albertisfu · 2024-12-24T01:40:23Z

juriscraper/pacer/email.py

+        #     these are usually 3 letters long. However, we want to keep some special acronyms
+        #     such as MOR (Merchant of Record?)
+        # - "NEF: " placeholder
+        component_regex = r"((?!-MOR)(\-[A-Z]{2,}))|(\-[a-z]{2,})|(NEF:? )"


I found a couple of examples where removing suffixes outside the exceptions listed here can be problematic:

For instance, this case where the subject contains:
Order -Non-motion related-

But it’s being parsed as:
Order -Non related-

pawd_1.txt

Or in the same case:
Motion to Withdraw/Dismiss Document -bk-

I couldn’t find an email example for this one, but I modified a subject and tested its parsing. The result was:
Motion to Withdraw/Dismiss Document

So this also seems difficult in determining when these suffixes should be removed or which ones should remain as exceptions. Perhaps we should just consider adding exceptions as we encounter them?

This seems easier maybe keeping the suffixes in parentheses works?

cacb.txt

ORDER to continue/reschedule hearing (BNC-PDF)

Currently the description is being parsed as:
ORDER to continue/reschedule hearing (BNC )

albertisfu · 2024-12-24T01:41:28Z

juriscraper/pacer/email.py

+        component_regex = r"((?!-MOR)(\-[A-Z]{2,}))|(\-[a-z]{2,})|(NEF:? )"
+        if self.court_id in ["paeb", "pamb"]:
+            # keeps the "Chapter ..." description on the short description
+            chapter_regex = r"(C[Hh][- ]?(13|7|9|11))|(C[hH][\s-]*$)"


Without looking too much I was able to find a couple of examples where the removal of Chapter... is not compatible.

For instance this one:
cacb_2.txt

You can see that in the Docket history report, many entries retain "Chapter" in their short descriptions.

Another example comes from pawb:

It seems difficult to determine when Chapter... should be removed. I’m also not sure if it's a consistent behavior across courts. If it should sometimes be removed and sometimes not in the same court that's a problem unless we can found a different pattern to decide. However, if it is consistent, we should look for more courts where it should be retained and add them to the condition.

grossir force-pushed the recap_email_bankruptcy_short_description branch from 1369764 to ecfc625 Compare December 17, 2024 05:23

grossir requested a review from mlissner December 17, 2024 15:02

grossir assigned mlissner Dec 17, 2024

grossir mentioned this pull request Dec 17, 2024

feat(pacer.email): multi docket NEF bankruptcy short description #1013

Closed

mlissner reviewed Dec 17, 2024

View reviewed changes

grossir commented Dec 17, 2024

View reviewed changes

fix(recap.email): refine bankruptcy short_description parsing

94b2d60

mlissner assigned albertisfu and unassigned mlissner Dec 18, 2024

mlissner requested a review from albertisfu December 18, 2024 17:59

grossir and others added 2 commits December 18, 2024 15:08

fix(pacer.email): improve bankruptcy short description

1e0dbe9

Keep "Chapter ..." string in paeb and pamb courts

[pre-commit.ci] auto fixes from pre-commit.com hooks

71281ae

for more information, see https://pre-commit.ci

grossir commented Dec 18, 2024

View reviewed changes

albertisfu reviewed Dec 24, 2024

View reviewed changes

albertisfu assigned grossir and unassigned albertisfu Dec 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pacer.email): improve bankruptcy short description parsing #1276

feat(pacer.email): improve bankruptcy short description parsing #1276

grossir commented Dec 17, 2024

mlissner left a comment

grossir left a comment

grossir Dec 17, 2024

grossir Dec 17, 2024

grossir Dec 17, 2024

grossir Dec 17, 2024

grossir Dec 17, 2024

grossir Dec 17, 2024

grossir Dec 17, 2024

grossir Dec 17, 2024

grossir Dec 17, 2024

grossir Dec 17, 2024

mlissner commented Dec 18, 2024

grossir commented Dec 18, 2024

grossir Dec 18, 2024

grossir Dec 18, 2024

grossir commented Dec 18, 2024

albertisfu left a comment

albertisfu Dec 24, 2024

albertisfu Dec 24, 2024

albertisfu Dec 24, 2024

feat(pacer.email): improve bankruptcy short description parsing #1276

Are you sure you want to change the base?

feat(pacer.email): improve bankruptcy short description parsing #1276

Conversation

grossir commented Dec 17, 2024

mlissner left a comment

Choose a reason for hiding this comment

grossir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlissner commented Dec 18, 2024

grossir commented Dec 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grossir commented Dec 18, 2024

albertisfu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment