Distribute binary wheels for Windows on PyPI

[birdcolour]

Hi there 👋

I've found my way here installing Fiona as a prerequisite for Geopandas on my work Windows machine. As a quick search of the issues reveals, I'm not the first to notice that there are no binary wheels available on PyPI for Windows. Rather than complain and/or not read the install instructions, I thought it would be more productive to formalise and track this feature request.

It's not clear to me what the specific roadblocks to this goal are, as the README suggests that development towards this goal is being made:

Binary installers are [...] coming eventually to PyPI

The Appveyor CI build [...] produces a binary wheel for successful builds [...] for [...] an unstable development version.

If there are specific tasks I or others can help with, please elaborate! If on the other hand, I've misread the situation and this isn't something that you want to support, please feel free to close this issue explaining why that is the case.

[sgillies]

Hi, @birdcolour. I've moved this to a discussion because I don't think it's actionable quite yet.

Fiona is, first and foremost, a software code project. It's not a software packaging or distribution project. We do publish some binaries, but their intent is to help make testing easier for developers of software that depend on this one. Historically, these developers have been working mainly on Linux and macOS, and so those platforms got more attention. Windows wheels would be nice to have, for sure, but this project doesn't have a ton of Windows experience and I'm not sure we get a lot back in return for learning how to put reliable Windows wheels out there. I could be wrong about this, but my feeling is based on 20 years of experience in open source.

The project does have Appveyor builds, but they are a bit neglected and haven't passed in a while. If I remember correctly, they use the "GISInternals" binaries at https://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries#Windows.

We've never had Windows wheel building at https://github.com/sgillies/fiona-wheels. It is a lot of work to keep the wheels building on Linux and macOS. I don't have the time or expertise to begin for Windows, and I'm not sure where to begin.

For the time being, I think it's best for Windows users to use packages from conda-forge. The conda-forge project is a software packaging project, not a software code project. You can even sponsor them with 💰 to assist them in their mission.

Lastly, it's my understanding that GeoPandas is going to replace its dependency on Fiona with a new package. If we do nothing and wait for a while, I think your problem will resolve itself.

I'm sorry I don't have good news for you. I hope this helps explain the situation.

[jorisvandenbossche]

Lastly, it's my understanding that GeoPandas is going to replace its dependency on Fiona with a new package. If we do nothing and wait for a while, I think your problem will resolve itself.

While we are indeed planning to have other options, just a note that pyogrio is in a very similar situation as fiona (it's basically a vectorized version of fiona, for the rest it uses the same building and dependency stack). We actually don't have wheels yet, and to add wheels, we will most likely mimic what fiona is currently doing. So moving to pyogrio will not resolve anything related to this out of the box.

(the idea is also to rely on a pure python package for some of the most used file formats, like pyshp, but that will never fully replace fiona or pyogrio)

---

That said, I have been experimenting a bit with vcpkg to build windows wheels, but nothing successful yet. If that would turn out to be a viable option for pyogrio, we should also be able to port that to the fiona wheel building.

[birdcolour]

Thanks for the clear response @sgillies! I 100% appreciate that packaging is a tedious and thankless task, and I don't doubt that the return on investment for such dedicated windows support doesn't really justify the amount of work you would need to put in. For what it's worth, I too would much rather be developing on/for linux/mac, but work dictates otherwise (that I'm leaving said job today is merely a coincidence 😉).

It sounds like from @jorisvandenbossche input that this problem does need to be solved somewhere. I certainly can't promise anything, but I might be able to find some time to hack on this problem in the next week or so, chiefly because I do think it is worthwhile making the geopandas installation process as painless as possible, but also because I'm not a huge fan of the conda approach in general. In this case, where would my efforts be best directed? Fiona or pyogrio? I didn't find building Fiona from source on Windows too tricky, so - famous last words - how hard can it be?

[jorisvandenbossche]

I might be able to find some time to hack on this problem in the next week or so, ... In this case, where would my efforts be best directed? Fiona or pyogrio? I didn't find building Fiona from source on Windows too tricky, so - famous last words - how hard can it be?

@birdcolour Thanks for the offer! I mentioned above that I had been experimenting a bit with vckg. I now was able to wrap up those experiments, and actually produced a wheel locally. See geopandas/pyogrio#48 for details. A first useful step would be to test this wheel to ensure it is properly working, but then helping to move this workflow to an automated github actions setup would be very useful.
This is targeting pyogrio, but as mentioned above, pyogrio and fiona have an almost identical setup, so if we get it working for pyogrio, it should be straightforward to port to fiona as well.

[jorisvandenbossche]

FYI I have some initial success at building a windows wheel on github actions (see geopandas/pyogrio#49)

[adamjstewart]

FWIW, rtree has Windows wheels. The CI code to do this doesn't look too bad. Maybe @hobu can comment on how easy/hard it is to get this working.

[hobu]

Fiona wheels is a huge undertaking because GDAL and all of its dependencies are a huge undertaking. Rtree wheels is one simple libspatialindex library that has no dependencies. That itself has been painful, if you look at our commit logs. Personally, I feel as if wheels are the right answer to the wrong question, and they put a huge burden on proprietors of projects that depend on native libraries to become multi-arch packagers using compiler configuration and settings tools that are the equivalent of a butter knife and a plastic bag (you can't stab yourself, so just put the bag over your head).

I totally understand wheels are convenient, but IMO that convenience should be afforded to customers, not users.

[jorisvandenbossche]

Another heads up that we managed to get to a quite satisfying solution in pyogrio using vcpkg + cibuildwheel for all three platforms (see https://github.com/geopandas/pyogrio/blob/b78e0a66260f093e33f9b41e95c83d60b93f88b1/.github/workflows/release.yml#L45-L137), and I think it should be quite straightforward to port this to fiona.

[adamjstewart]

Shapely also has Windows wheels and might be a better starting point since it shares many of the same developers. Shapely uses cibuildwheel as well: https://github.com/shapely/shapely/blob/main/.github/workflows/release.yml

[jorisvandenbossche]

I can ensure that pyogrio is a better starting point :) It has the exact same dependency and build chain as fiona (GDAL + cython wrappers). Shapely is much simpler because GEOS is a simple C++ library without additional dependencies.

(but anyway, my work in pyogrio was also based on what we did in Shapely using cibuildwheel, so it's already based on that :))

[sgillies]

@jorisvandenbossche thanks to your excellent work on pyogrio wheels, we have similar ones for Fiona now: #1117 (comment).