Model file export

[omer-candan]

Write models to .mps files

We can export linear solver models and get a string as a result. However, when working with very large models, the process (.NET in my case) fails when the export method is called, due to string type's size limitation. In our use case, we don't actually need the exported model in memory, we write the string to a file. A method that directly writes to a file helps us overcome the limitation.
The file write method has been added by @lperron, which simplifies this PR.

[lperron]

this is interesting. I cannot use it as is. We cannot use iostream. The complex path is to implement gzip reading/writing in the File interface, which is a pain.

[omer-candan]

this is interesting. I cannot use it as is. We cannot use iostream. The complex path is to implement gzip reading/writing in the File interface, which is a pain.

Thanks @lperron . My first attempt was to not use iostream. I had to duplicate ExportModelAsLpFormat, AppendComments and AppendConstraint functions to directly write to gz files. I guess that would also not be OK, because of too much duplication.
Did you mean something similar?

[lperron]

So, the best way to integrate it to add a field in the File class (ortools/base/file.h) that indicates if the file is a gzip one.
In that case, in the relevant methods, we need to duplicate the code to use gzip method instead of the C FILE* methods.

For instance, the MPS reader uses the FileLine class, which is turns call File::Read() raw API.

On the MPS writer, I would make sure that we flush the output string to file regularly, and that this flush is done by the File API, which internally would use the gzip API.

Am I clear ?

[omer-candan]

So, the best way to integrate it to add a field in the File class (ortools/base/file.h) that indicates if the file is a gzip one. In that case, in the relevant methods, we need to duplicate the code to use gzip method instead of the C FILE* methods.

For instance, the MPS reader uses the FileLine class, which is turns call File::Read() raw API.

On the MPS writer, I would make sure that we flush the output string to file regularly, and that this flush is done by the File API, which internally would use the gzip API.

Am I clear ?

Yes, thanks again! I'll try my best to work on those parts.

[lperron]

Please sync with main. You will have a conflict.
I have implemented MpModelProtoExporter::WriteModelToMpsFile() and hooked it with model_builder (python, java, .NET).
The implementation is not that robust to very large models, but better than before.

The only missing is gzip support in File (and hooking to linear_solver C# if you want it).

[omer-candan]

Please sync with main. You will have a conflict. I have implemented MpModelProtoExporter::WriteModelToMpsFile() and hooked it with model_builder (python, java, .NET). The implementation is not that robust to very large models, but better than before.

The only missing is gzip support in File (and hooking to linear_solver C# if you want it).

@lperron Thank you so much. I resolved the conflicts and called your function from linear_solver. Made it optional to export to gzip with a bool parameter.
I had a difficult time adding compression part. Your method can write a 13GB(1.2GB compressed) model file I used in my tests in a single call, but it was not possible with the new gzip write (gzwrite). The compressed output files were cut off and usually not decompressable. Multiple write operations to work with chunks solved this problem. Is this an OK approach? If yes, I'd like to get your opinion on chunk size and compression level (1=fastest now) that I chose.

[omer-candan]

I'll resolve the conflicts

[omer-candan]

Thank you @Mizux for including this in the next release candidate list.

Thank you @lperron for including file write operations for other language wrappers. File write is already huge for us, being able to compress will be a real nice to have too. I updated my branch and it only has gzip file compression now.
I had tested the previous version of my PR and if you approve of these changes I can test this last changeset again.

[Mizux]

"Error while closing the file" or "Unable to close the file"

[omer-candan]

Thanks 👍 . I chose "Unable to close the file" to match with the error message thrown when opening a file.