Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pickling functions defined in doctests fail #101

Open
Erotemic opened this issue Aug 22, 2021 · 1 comment
Open

Pickling functions defined in doctests fail #101

Erotemic opened this issue Aug 22, 2021 · 1 comment

Comments

@Erotemic
Copy link
Owner

New ubelt futures test failed when trying to run in xdoctest:

    >>> # xdoctest: +SKIP
    >>> # Note: while this works in IPython, this does not work when running
    >>> # in xdoctest. 
    >>> # xdoctest: +REQUIRES(module:timerit)
    >>> # Does my function benefit from parallelism?
    >>> def my_function(arg1, arg2):
    ...     return (arg1 + arg2) * 3
    >>> #
    >>> def run_process(inputs, mode='serial', max_workers=0):
    ...     from concurrent.futures import as_completed
    ...     import ubelt as ub
    ...     # The executor interface is the same regardless of modes
    ...     executor = ub.Executor(mode=mode, max_workers=max_workers)
    ...     # submit returns a Future object
    ...     jobs = [executor.submit(my_function, *args) for args in inputs]
    ...     # future objects will contain results when they are done
    ...     results = [job.result() for job in as_completed(jobs)]
    ...     return results
    >>> # The same code tests our method in serial, thread, or process mode
    >>> import timerit
    >>> ti = timerit.Timerit(100, bestof=10, verbose=2)
    >>> # Setup test data
    >>> import random
    >>> rng = random.Random(0)
    >>> max_workers = 4
    >>> inputs = [(rng.random(), rng.random()) for _ in range(100)]
    >>> for mode in ['serial', 'process', 'thread']:
    >>>     for timer in ti.reset('mode={} max_workers={}'.format(mode, max_workers)):
    >>>         with timer:
    >>>             run_process(inputs, mode=mode, max_workers=max_workers)
    >>> print(ub.repr2(ti))

Error is:

    DOCTEST TRACEBACK
    concurrent.futures.process._RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
        obj = _ForkingPickler.dumps(obj)
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
        cls(buf, protocol).dump(obj)
    _pickle.PicklingError: Can't pickle <function my_function at 0x7f7f27f5e430>: attribute lookup my_function on ubelt.util_futures failed
    """
    
    
    The above exception was the direct cause of the following exception:
    
    
    Traceback (most recent call last):
    
      File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 27, abs: 37, in <module>
        >>>             run_process(inputs, mode=mode, max_workers=max_workers)
    
      File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 14, abs: 24, in run_process
        ...     results = [job.result() for job in as_completed(jobs)]
    
      File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 14, abs: 24, in <listcomp>
        ...     results = [job.result() for job in as_completed(jobs)]
    
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/_base.py", line 432, in result
        return self.__get_result()
    
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
        raise self._exception
    
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
        obj = _ForkingPickler.dumps(obj)
    
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
        cls(buf, protocol).dump(obj)
    
    _pickle.PicklingError: Can't pickle <function my_function at 0x7f7f27f5e430>: attribute lookup my_function on ubelt.util_futures failed
    DOCTEST REPRODUCTION
    CommandLine:
        python -m xdoctest /home/joncrall/code/ubelt/ubelt/util_futures.py __doc__:0

=== Failed tests ===
python -m xdoctest /home/joncrall/code/ubelt/ubelt/util_futures.py __doc__:0
=== 1 failed, 6 passed in 0.14 seconds ===
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/process.py", line 104, in _python_exit
Process ForkProcess-4:
Process ForkProcess-2:
Process ForkProcess-1:
Process ForkProcess-3:
@Erotemic
Copy link
Owner Author

This is related to the issue I just saw in scriptconfig.

The fix is that we need to allow xdoctest to create a dummy module scope for it to execute its code in. See: https://stackoverflow.com/questions/5122465/can-i-fake-a-package-or-at-least-a-module-in-python-for-testing-purposes

Not exactly sure how to do this yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant