OS X package contains random X11 headers

[pkgw]

(This is the same as anaconda-issues #600.)

I'm not sure if this is a bug or a feature, but the OS X package of tk includes some X11 header files that, in my opinion, almost surely don't belong:

  "files": [
    "bin/tclsh8.5", 
    "bin/wish8.5", 
    "include/X11/X.h", 
    "include/X11/Xatom.h", 
    "include/X11/Xfuncproto.h", 
    "include/X11/Xlib.h", 
    "include/X11/Xutil.h", 
    "include/X11/cursorfont.h", 
    "include/X11/keysym.h", 
    "include/X11/keysymdef.h", 
    "include/X11/xbytes.h", 
    "include/tcl.h", 
  ...

This causes problems for me because I'm working on a "modern Xorg stack" package that tries to provide a consistent X11 library environment for building other software, and it needs Python (hence tk) to build. So tk's X11 headers supersede mine. The mix-and-match of header files causes all sorts of bad news downstream.

I'll note that the linux-64 package of tk does not include these files.

[jjhelmus]

I agree that it seems strange that these headers are included in the OS X package but looking at the Makefile.in file, this behavior seems to be intentional. A check is made to see if the X11 headers are available on the system and if not the ones included with tk are installed along with the Tk headers.

I'm worried that removing these may break downstream packages which expect these headers to be present. Looking at other other OS X based tk installs, they all seem to include the X11 headers. For example on my Mac, Apple is including these files in the Tk framework.

[pkgw]

In that case, I would propose that the OSX version of the tk package depend on the modern-xorg-stack package that I have submitted, which would provide the X11 headers in a more sensible place.

Admittedly, this would be a fairly big change: you'd be setting conda-forge on the path of providing its own X11 libraries everywhere. Which I think would be good! But it would be big.

[pkgw]

A status update here. For context: I have some packages that require full X11 client libraries on OSX and at first I was hoping that I'd be able to build and deploy them under the current state of affairs. But I'm now convinced that, no, for them to work, conda-forge needs to provide its own set of X11 client libraries. We discussed this in the January videoconference and there was consensus that this was a direction in which people were comfortable going. So I've begun submitting packages for the X.org X11 libraries, as has been discussed a bit in the modern-xorg-stack PR linked in my previous comment.

As for tk: I've investigated and as far as I can tell, the X11 headers are distributed as a hack: the tk headers require X11 types, but they are not provided on OSX, so the tk maintainers decided to just bundle copies. I'm not sure exactly how the build logic works, but I'd hope that it wouldn't install its copies unless no other versions were available. So, once the X.org client libraries are all incorporated as conda-forge packages, I would like to add a small number of packages like xorg-xproto as dependencies for tk. In a certain sense they're only build-time dependencies, but maybe it would be better to list them as run-time dependencies as well — anyone wishing to develop against tk will need them to be installed too. (It would be nice if conda had a concept of "devel" packages like most other packaging systems.)

[pkgw]

Pull request #17 uses the newly-available xorg-libx11 package to implement the plan described above.

[jakirkham]

Also figured I'd mention this comment. That should be a good intermediate solution if switching this over to X11 immediately feels like too big of a change ATM.

[pkgw]

Ah, that's very helpful! But I'll emphasize that always_include_files (the technique referenced in @jakirkham's comment) is not an adequate long-term fix because if your package has both xorg-libx11 and tk as dependencies, you don't know whose headers are going to end up "winning" in the environment setup process. (Unless the installation is more deterministic than I'd guess?) Which wouldn't be such a big deal since we probably don't have that many packages that use tk, but python brings it in, and lots of things have python as a build-time dependency.

[jakirkham]

Completely agree @pkgw.

Not advocating it as a long term fix either. More as a way of letting X.org libraries to continue to be added ATM without letting this Tk issue become a total blocker. We certainly will need to start embracing X.org instead of say yum_requirements.txt and standardizing Mac and Linux a bit w.r.t. X.org libraries. This just gives people more time to mull over what needs to happen as we progress.

[beckermr]

This issue is causing me grief and it appears we have a lot more xorg stuff in conda-forge. What is the path to making a useful PR here?

[pkgw]

@beckermr Good question. It's clear that there's a lot of desire to maintain good alignment with mainline Anaconda, which does not provide its own X packages and uses the CDT model for Linux packages that do depend on X (among other things). I think it would be very hard to persuade folks to introduce a new dep for tk since it is a foundational package in the ecosystem.

Thus far, I've always been able to get things working under the status quo. With a concerted effort I'm sure that folks could converge on a more self-consistent and sensible situation, but empirically it doesn't seem to have risen up anybody's priority queue. This seems like an area where we're going to be limping along awkwardly for a while :-/

[beckermr]

Got it. Can we ask the upstream tk devs to change this maybe instead of us trying to patch around it?

[rtburns-jpl]

It seems to me that the headers are overwritten by whichever package is installed last, so I was able to work around this by force-reinstalling the xorg-* packages in my build recipe. Very kludgy solution, but just thought I'd share in case this is causing headaches for anyone else.