[WORKSPACE, etc.] New repository rule to compute local Python repo.

This replaces the hard-coded `python.BUILD` BUILD file with an active
step that runs the system's active Python interpreter in order to
determine the system's Python installation's include paths.

Author: tkoeppe

.github/workflows/precommit.yml

@@ -38,6 +38,7 @@ jobs:
           sudo apt-get install gettext python2-dev python3-dev libsdl2-dev libosmesa6-dev python-numpy-dev python3-numpy-dev python-pil python3-pil python-enum34
           sudo find /usr -type f -name Python.h -o -name arrayobject.h -o -name functional -o -name -filesystem
           sudo ln -s /usr/lib/llvm-10/include/c++/v1 /usr/include/c++/v1
+          sed -i -e 's/py_version = "PY3"/py_version = "PY2AND3"/' WORKSPACE
       - name: build
         run: |
           bazel --bazelrc=.precommit.bazelrc build ${{matrix.cfg.config}} --copt=-Wno-sign-compare //...

BUILD

@@ -1027,7 +1027,7 @@ cc_binary(
     ],
     deps = [
         ":dmlablib",
-        "@python_system//:python",
+        "@python_system//:python_headers",
     ],
 )
 

WORKSPACE

@@ -1,6 +1,7 @@
 workspace(name = "org_deepmind_lab")
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+load("@//:python_system.bzl", "python_repo")
 
 http_archive(
     name = "com_google_googletest",
@@ -158,8 +159,7 @@ new_local_repository(
     path = "/usr",
 )
 
-new_local_repository(
+python_repo(
     name = "python_system",
-    build_file = "@//bazel:python.BUILD",
-    path = "/",
+    py_version = "PY3",
 )

bazel/python.BUILD

@@ -68,10 +68,7 @@ If Python 3 support is not required, omit the packages that mention `python3`.
 
 3. [Clone or download *DeepMind Lab*](https://github.com/deepmind/lab).
 
-4. If necessary, edit `python.BUILD` according to the [Python
-   instructions](#python-dependencies) below.
-
-5. Build *DeepMind Lab* and run a random agent. (Use the `-c opt` flag to enable
+4. Build *DeepMind Lab* and run a random agent. (Use the `-c opt` flag to enable
    optimizations.)
 
    ```shell
@@ -97,9 +94,9 @@ arguments. Run `bazel run :random_agent -- --help` to see those.
 
 *DeepMind Lab* does not include every dependency hermetically. In particular,
 Python is not included, but instead it must already be installed on your system.
-This means that depending on the details of where that library is installed, you
-may need to adjust the Bazel build rules in
-[`python.BUILD`](../../bazel/python.BUILD) to locate it correctly.
+Our Bazel workspace includes a mechanism to discover the location of the
+system's Python paths automatically by running the `python2` and `python3`
+interpreters. Additionally, NumPy must be available on your system, too.
 
 Bazel can build Python code using either Python 2 or Python 3. The default is
 Python 3, but each individual `py_binary` and `py_test` target can specify the
@@ -108,57 +105,10 @@ desired version using the
 argument. The build rules need to make the local installation path of correct
 version of Python available.
 
-If you only intend to use one of the two versions (e.g. on an older system where
-Python 3 with NumPy is not available), you only need to provide paths for that
-version; however, the codebase includes tests that run under both Python 2 and
-Python 3.
-
 The default build rules should work for Debian and Ubuntu. They use Bazel's
 [configurable attributes](https://docs.bazel.build/versions/master/be/common-definitions.html#configurable-attributes)
 to provide paths for Python 2 and Python 3, respectively, based on which version
-is required during a particular build. Note that paths in the build rules are
-relative to the root path specified in the [`WORKSPACE`](../../WORKSPACE) file
-(which is `"/usr"` by default).
-
-Python requires two separate dependencies: The CPython extension API, and NumPy.
-If, say, NumPy is installed in a custom location, like it is on SUSE Linux and
-RedHat Linux, you need to add the files from that location and set an include
-search path accordingly. For example:
-
-```python
-cc_library(
-    name = "python",
-    hdrs = select(
-        {
-            "@bazel_tools//tools/python:PY2": glob([
-                "include/python2.7/*.h",
-                "lib64/python2.7/site-packages/numpy/core/include/**/*.h",
-            ]),
-            "@bazel_tools//tools/python:PY3": glob([
-                "include/python3.6m/*.h",
-                "lib64/python3.6/site-packages/numpy/core/include/**/*.h",
-            ]),
-        },
-        no_match_error = "Internal error, Python version should be one of PY2 or PY3",
-    ),
-    includes = select(
-        {
-            "@bazel_tools//tools/python:PY2": [
-                "include/python2.7",
-                "lib64/python2.7/site-packages/numpy/core/include",
-            ],
-            "@bazel_tools//tools/python:PY3": [
-                "include/python3.6m",
-                "lib64/python3.6/site-packages/numpy/core/include",
-            ],
-        },
-        no_match_error = "Internal error, Python version should be one of PY2 or PY3",
-    ),
-    visibility = ["//visibility:public"],
-)
-```
-The outputs of `rpm -ql python` and `rpm -ql python-numpy-devel` might be
-helpful to find the right include directories on Red-Hat-like systems.
+is required during a particular build.
 
 If you have installed NumPy locally via PIP and would like to use the *DeepMind
 Lab* PIP module, then you should build the module against the version of NumPy

docs/users/build.md

@@ -0,0 +1,72 @@
+# Copyright 2021 DeepMind Technologies Limited.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# ============================================================================
+
+"""Generates a local repository that points at the system's Python installation."""
+
+_BUILD_FILE = '''# Description:
+#   Build rule for Python
+
+exports_files(["defs.bzl"])
+
+cc_library(
+    name = "python_headers",
+    hdrs = select({
+        "@bazel_tools//tools/python:PY2": glob(["python2/**/*.h", "numpy2/**/*.h"]),
+        "@bazel_tools//tools/python:PY3": glob(["python3/**/*.h", "numpy3/**/*.h"]),
+    }),
+    includes = select({
+        "@bazel_tools//tools/python:PY2": ["python2", "numpy2"],
+        "@bazel_tools//tools/python:PY3": ["python3", "numpy3"],
+    }),
+    visibility = ["//visibility:public"],
+)
+'''
+
+_GET_PYTHON_INCLUDE_DIR = """
+import sys
+from distutils.sysconfig import get_python_inc
+from numpy import get_include
+sys.stdout.write("{}\\n{}\\n".format(get_python_inc(), get_include()))
+""".strip()
+
+def _python_repo_impl(repository_ctx):
+    """Creates external/<reponame>/BUILD, a python3 symlink, and other files."""
+
+    repository_ctx.file("BUILD", _BUILD_FILE)
+
+    if repository_ctx.attr.py_version in ["PY2", "PY2AND3"]:
+        result = repository_ctx.execute(["python2", "-c", _GET_PYTHON_INCLUDE_DIR])
+        if result.return_code:
+            fail("Failed to run local Python2 interpreter: %s" % result.stderr)
+        pypath, nppath = result.stdout.splitlines()
+        repository_ctx.symlink(pypath, "python2")
+        repository_ctx.symlink(nppath, "numpy2")
+
+    if repository_ctx.attr.py_version in ["PY3", "PY2AND3"]:
+        result = repository_ctx.execute(["python3", "-c", _GET_PYTHON_INCLUDE_DIR])
+        if result.return_code:
+            fail("Failed to run local Python3 interpreter: %s" % result.stderr)
+        pypath, nppath = result.stdout.splitlines()
+        repository_ctx.symlink(pypath, "python3")
+        repository_ctx.symlink(nppath, "numpy3")
+
+python_repo = repository_rule(
+    implementation = _python_repo_impl,
+    configure = True,
+    local = True,
+    attrs = {"py_version": attr.string(default = "PY2AND3", values = ["PY2", "PY3", "PY2AND3"])},
+)