Handle UTF encoded scene description files

[FilipeXavierGama]

Hi,

I have an odd (systematic nevertheless) problem after converting pbrt-v3 to pbrt-v4 files.
First of all, I have used the following command for conversion:
$ pbrt --upgrade old.pbrt > new.pbrt

Then, if I try to use "new.pbrt" as the input argument to "pbrt", $ pbrt new.pbrt, I obtain the following errors:
End of files before "WorldBegin"

Apparently, the pointer to the input file is not working as it should for some unknown reason, returning "end-of-file" when there is error-free data to read inside the input file.

However, if I copy the data manually from "new.pbrt" to another new empty file (for example "new_v2.pbrt"), pbrt works just fine using $ pbrt new_v2.pbrt

-> My guess is that there is some issue while creating and converting a pbrt-v3 file to a pbrt-v4 file.

Test case scenario:

• Create a "pbrt-v3" input file (named "Scene_pbrt3.pbrt" in the attachment) with the default scene described in https://www.pbrt.org/fileformat-v3 under Example of a pbrt file

• Convert the "pbrt-v3" file to a "pbrt-v4" file (named "Scene_pbrt4.pbrt" in the attachment) using $ pbrt --upgrade Scene_pbrt3.pbrt > Scene_pbrt4.pbrt

• Create a new empty file called "Scene_pbrt4_2.pbrt" (attached) and copy manually the content from "Scene_pbrt4.pbrt" to "Scene_pbrt4_2.pbrt".

-> From the test case scenario above, $ pbrt Scene_pbrt4.pbrt fails and $ pbrt Scene_pbrt4_2.pbrt works fine.

I'm running PBRT-v4 with GPU support on Windows 10, RTX 2080 Super Mobile, Cuda Toolkit 11.2.0, Optix SDF 7.3.0.

TestCaseScenario.zip

[pbrt4bounty]

Hi.. After a couple of checks, seems that the problem is the text codification format in the generated pbrt4.pbrt file. I forced to change this format to ASCII, and the file is now properly rendered with pbrt 4.

This are the warning..

[pbrt4bounty]

I think the problem maybe is the default encoding text files from your OS. Here, under Windows, the --upgrade parameter generated a right file format and is rendered Ok.

..\pbrt4bounty\bin>pbrt --upgrade pbrt3.pbrt >my_pbrt4.pbrt

..\pbrt4bounty\bin>pbrt my_pbrt4.pbrt
pbrt version 4 (built May 29 2021 at 21:28:28)
Copyright (c)1998-2021 Matt Pharr, Wenzel Jakob, and Greg Humphreys.
The source code to pbrt (but *not* the book contents) is covered by the Apache 2.0 License.
See the file LICENSE.txt for the conditions of the license.
Rendering: [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++]  (50.9s)

What do you think @mmp ??
Cheers..

[FilipeXavierGama]

I think the problem maybe is the default encoding text files from your OS. Here, under Windows, the --upgrade parameter generated a right file format and is rendered Ok.

..\pbrt4bounty\bin>pbrt --upgrade pbrt3.pbrt >my_pbrt4.pbrt

..\pbrt4bounty\bin>pbrt my_pbrt4.pbrt
pbrt version 4 (built May 29 2021 at 21:28:28)
Copyright (c)1998-2021 Matt Pharr, Wenzel Jakob, and Greg Humphreys.
The source code to pbrt (but *not* the book contents) is covered by the Apache 2.0 License.
See the file LICENSE.txt for the conditions of the license.
Rendering: [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++]  (50.9s)

What do you think @mmp ??
Cheers..

Thanks for checking this out. Looks like PBRT-v4 cannot deal with UTF-16LE encoding formats (and maybe that's ok!). Windows' Powershell creates UTF-16LE files by default while cmd shell uses the legacy encoding windows-1252 / UTF-8 by default. This means, that if I use the Powershell, I face the issue above. If I use "cmd", then everything is fine.

Cheers!

[mmp]

Nice detective work @pbrt4bounty! I've just pushed a small fix that at least issues a sensible error message if a UTF-16-encoded file is detected.

I've dug into this a bit and my thinking is that now that it's 2021, pbrt should stop assuming ASCII and should at least support UTF-8-encoded scene description files. That will take a bit of re-plumbing; std::string is used all over the place in the system, for example. I'll rename this issue to cover that work and will eventually take care of it...

[mmp]

Upon further education about Unicode, turns out that std::string is fine (given UTF8 and then taking things from there.)

In any case, as far as I can tell, pbrt now properly handles(*) UTF8, both in *.pbrt files as well as in command line arguments. So things like ObjectBegin "🚀🙏", or pbrt 🚒.pbrt all work, on Linux, OSX, and Windows. (More prosaically, non-ASCII character sets should now just work.)

(*) The two remaining exceptions are that 1) UTF8 filenames for PNGs don't work, but there is a lodepng PR that would make it possible to address that. There is also an issue in the ext/filesystem dependency that can cause trouble on Windows under some circumstances, also hopefully fixed soon.