-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tie Breaker Error #134
Comments
As the research paper points: "If it is not possible to break the tie reliably, our method returns no result. In a practical setting, this is preferred over returning an incorrect result." So, there is no issues with CleverCSV on this subject. The tool is working as expected. |
In what scenario would a none type response be better? In my use-case, tie breaker fails when there's two dialects where it may or may not have quotation marks. As in, it doesn't matter. a better approach would be returning which ever is highest score, both, or one.
|
If you are running CleverCSV in verbose mode, you can check dialects having highest scores. At this point, human intervention is needed, so manually select the one that suits your needs. |
Gotchya, that seems counter productive in my use-case specifically. I still think in such cases the response should be a list of possible dialects rather than none. Anyways marking as closed since it appears there isn't much interest here besides me. |
I have developed a mechanism for dialect detection in Python. The result is always a dialect, the first having the highest score. Then, the user dialect predefined ordering is predominant. Feel free to try it. https://content.iospress.com/articles/data-science/ds240062 |
will do! Thank you, interesting read so far. |
@0xSheller, here is the GitHub repo. |
I've been rooting out this problem for a few hours now and I've finally figured out where it's stemming from after concluding it's not in my implementation/code base.
Issue:
When detecting dialects, and two or more dialects have the exact same score, the tiebreaker returns NoneType.
See:
This specifically yields 2 dialects that go into the
break_ties_two
function. I'm unsure if this affects the other tiebreakers:break_ties_three
break_ties_four
Modifying the
detect
function to print results:Yields:
This tells me the tie breaker isn't doing it's job.
Further Analysis:
Ambiguous Tie Breakers: The function relies on several specific conditions to break ties between dialects, such as differences in quotechar, delimiter, or escapechar. If these conditions are not met, or if the distinctions are not sufficient to determine a clear winner (i.e., none of the predefined conditions apply or the conditions apply but lead to a tie), the function defaults to returning None.
Incomplete Handling for Non-Specific Cases: The current implementation does not cover cases where dialects differ in ways that are not specified or when the parsing results in identical outputs for the specific attributes being compared.
The text was updated successfully, but these errors were encountered: