FutureWarnings in LocalCluster not being raised as errors

[pentschev]

We intentionally raise DeprecationWarnings/FutureWarnings as errors in pytest, such that we're always notified when something will demand a change on our end. This works fine for warnings being thrown at the parent pytest process, but apparently the same doesn't happen for LocalCluster and children classes.

dask_cuda/tests/test_warning.py

import warnings

from distributed import Client, LocalCluster
from distributed.utils_test import gen_test


def test_future_warning_as_error():
    warnings.warn(
        "This should error",
        FutureWarning,
    )


@gen_test(timeout=120)
async def test_future_warning_as_error_local_cluster():
    async with LocalCluster(
        n_workers=1,
        scheduler_port=0,
        silence_logs=False,
        dashboard_address=None,
        asynchronous=True,
        memory_target_fraction=False,
        memory_spill_fraction=False,
        memory_pause_fraction=False,
    ):
        pass


def test_future_warning_as_error_local_cluster_sync():
    with LocalCluster(
        protocol="tcp",
        dashboard_address=None,
        n_workers=4,
        threads_per_worker=5,
        processes=True,
        memory_target_fraction=False,
        memory_spill_fraction=False,
        memory_pause_fraction=False,
    ):
        pass

Running that will cause the warning in the pytest parent process to be raised as error, as we expect, but both sync and async versions of LocalCluster will raise FutureWarnings without any being raised as errors, see below.

output

=============================== test session starts ===============================
platform linux -- Python 3.9.18, pytest-7.4.2, pluggy-1.3.0 -- /opt/conda/envs/test/bin/python3.9
cachedir: .pytest_cache
rootdir: /repo
configfile: pyproject.toml
plugins: cov-4.1.0
collected 3 items

tests/test_warning.py::test_future_warning_as_error FAILED
tests/test_warning.py::test_future_warning_as_error_local_cluster /opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_target_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.target instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_spill_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.spill instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_pause_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.pause instead
  warnings.warn(
2023-09-26 11:20:54,926 - distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:34133
2023-09-26 11:20:54,926 - distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:34133
2023-09-26 11:20:54,926 - distributed.worker - INFO -           Worker name:                          0
2023-09-26 11:20:54,926 - distributed.worker - INFO -          dashboard at:            127.0.0.1:38401
2023-09-26 11:20:54,926 - distributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:45147
2023-09-26 11:20:54,926 - distributed.worker - INFO - -------------------------------------------------
2023-09-26 11:20:54,926 - distributed.worker - INFO -               Threads:                         80
2023-09-26 11:20:54,926 - distributed.worker - INFO -                Memory:                   0.98 TiB
2023-09-26 11:20:54,926 - distributed.worker - INFO -       Local Directory: /tmp/dask-scratch-space/worker-7x4x64mi
2023-09-26 11:20:54,927 - distributed.worker - INFO - -------------------------------------------------
2023-09-26 11:20:55,834 - distributed.worker - INFO - Starting Worker plugin shuffle
2023-09-26 11:20:55,835 - distributed.worker - INFO -         Registered to:      tcp://127.0.0.1:45147
2023-09-26 11:20:55,836 - distributed.worker - INFO - -------------------------------------------------
2023-09-26 11:20:55,837 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45147
2023-09-26 11:20:55,889 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34133. Reason: nanny-close
2023-09-26 11:20:55,891 - distributed.core - INFO - Connection to tcp://127.0.0.1:45147 has been closed.
2023-09-26 11:20:55,893 - distributed.nanny - INFO - Worker closed
PASSED
tests/test_warning.py::test_future_warning_as_error_local_cluster_sync /opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_target_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.target instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_spill_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.spill instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_pause_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.pause instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_target_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.target instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_spill_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.spill instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_pause_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.pause instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_target_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.target instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_spill_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.spill instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_pause_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.pause instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_target_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.target instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_spill_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.spill instead
  warnings.warn(
/opt/conda/envs/test/lib/python3.9/site-packages/distributed/worker_memory.py:493: FutureWarning: Parameter memory_pause_fraction has been deprecated and will be removed in a future version; please use dask config key distributed.worker.memory.pause instead
  warnings.warn(
PASSED/opt/conda/envs/test/lib/python3.9/site-packages/coverage/inorout.py:507: CoverageWarning: Module dask_cuda was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/opt/conda/envs/test/lib/python3.9/site-packages/coverage/control.py:883: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
WARNING: Failed to generate report: No data to report.

/opt/conda/envs/test/lib/python3.9/site-packages/pytest_cov/plugin.py:298: CovReportWarning: Failed to generate report: No data to report.

  self.cov_controller.finish()


==================================== FAILURES =====================================
__________________________ test_future_warning_as_error ___________________________

    def test_future_warning_as_error():
>       warnings.warn(
            "This should error",
            FutureWarning,
        )
E       FutureWarning: This should error

tests/test_warning.py:8: FutureWarning
----------- generated xml file: /repo/test-results/junit-dask-cuda.xml ------------

---------- coverage: platform linux, python 3.9.18-final-0 -----------

================================ slowest durations ================================
2.13s call     dask_cuda/tests/test_warning.py::test_future_warning_as_error_local_cluster
1.93s call     dask_cuda/tests/test_warning.py::test_future_warning_as_error_local_cluster_sync
0.00s setup    dask_cuda/tests/test_warning.py::test_future_warning_as_error
0.00s teardown dask_cuda/tests/test_warning.py::test_future_warning_as_error_local_cluster_sync
0.00s teardown dask_cuda/tests/test_warning.py::test_future_warning_as_error_local_cluster
0.00s setup    dask_cuda/tests/test_warning.py::test_future_warning_as_error_local_cluster
0.00s setup    dask_cuda/tests/test_warning.py::test_future_warning_as_error_local_cluster_sync
0.00s teardown dask_cuda/tests/test_warning.py::test_future_warning_as_error
0.00s call     dask_cuda/tests/test_warning.py::test_future_warning_as_error
============================= short test summary info =============================
FAILED tests/test_warning.py::test_future_warning_as_error - FutureWarning: This should error
=========================== 1 failed, 2 passed in 5.54s ===========================

I don't know if there's anything we can do in pytest so that it will capture warnings in subprocesses as well, but that would be the case we truly want to test.