-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Path parsing error in ParquetWriter on Windows #3029
Comments
IIUC, cc @dlvenable as this issue is related to #1376 |
I can remote it when creating Path in example above (first line) but ParquetWriter will add it again. Before calling |
Hello, |
This is a regression introduced by #1376. I don't think we can work around it easily. Should we revert that commit? @ggershinsky @gszadovszky? |
I don't think we should revert the original change. |
@martin-frydl Have you tried to use |
Could you please have a try on #3038? @martin-frydl |
@wgtmac Yes, that fixes the problem. Thank you. |
@wgtmac thank you for the fix. It will help us. |
Describe the bug, including details regarding any error messages, version, and platform.
Hello,
I've found problem with parsing Windows paths in ParquestWriter since version 1.14.2. Given following code:
the exception is thrown:
The problem is caused by change in ParquetWriter:392. Here used to be
new Path(path)
where Path is from org.apache.hadoop.fs (see here) but now there isPaths.get(path)
from java.nio.file. The problem is that path is in fact URI, i.e. something likefile:/...
, but java.nio.file.Path does not expect URI. It probably does not work correctly also on Linux but there is no exception since ":" is OK in path there.Please note that the example code does not fully work. The code is part of larger application with complex initialization so right now I don't have the correct schema for the writer. Fortunately that exception is thrown before bad schema error.
Component(s)
Core
The text was updated successfully, but these errors were encountered: