Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap #14

Open
houdinisparks opened this issue Jul 28, 2018 · 7 comments
Open

AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap #14

houdinisparks opened this issue Jul 28, 2018 · 7 comments
Assignees
@borgstrom

Comments

@houdinisparks
Copy link

Hi,

when i try to initialise the KinesisProducer object, it throws this error.

self.kproducer = KinesisProducer(stream_name=produce_stream_name)

Traceback:

('2018-07-28 23:58:19,660 - botocore.loaders - MainThread - DEBUG - Loading JSON file: C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\botocore\data\_retry.json
('2018-07-28 23:58:19,662 - botocore.client - MainThread - DEBUG - Registering retry handlers for service: kinesis
Traceback (most recent call last):
  File "C:\Users\662176\AppData\Roaming\JetBrains\PyCharm Community Edition 2017.3.3\helpers\pydev\pydevd.py", line 1668, in <module>
    main()
  File "C:\Users\662176\AppData\Roaming\JetBrains\PyCharm Community Edition 2017.3.3\helpers\pydev\pydevd.py", line 1662, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Users\662176\AppData\Roaming\JetBrains\PyCharm Community Edition 2017.3.3\helpers\pydev\pydevd.py", line 1072, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Users\662176\AppData\Roaming\JetBrains\PyCharm Community Edition 2017.3.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/662176/Documents/_Projects/_makerfaire/drone_showcase/pose_recognizer/src/webapp/app.py", line 27, in <module>
    consumer_stream_name="poserec_results")
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\src\poseapp\poseapp_kinesis.py", line 41, in __init__
    self.kproducer = KinesisProducer(stream_name=produce_stream_name)
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\kinesis\producer.py", line 142, in __init__
    max_size=max_size, boto3_session=boto3_session)
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\kinesis\producer.py", line 78, in __init__
    self.start()
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 55, in start
    self.process.start()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 75, in atexit
    recursively_shutdown(cls)
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 71, in recursively_shutdown
('2018-07-28 23:58:19,957 - offspring.process - MainThread - DEBUG - Shutting down <kinesis.producer.AsyncProducer object at 0x000002746D1E68D0>
    recursively_shutdown(klass)
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 74, in recursively_shutdown
    obj.shutdown()
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 85, in shutdown
    self.wait()
  File "C:\Users\662176\Documents\_Projects\_makerfaire\drone_showcase\pose_recognizer\venv\lib\site-packages\offspring\process.py", line 89, in wait
    self.process.join()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 123, in join
    assert self._popen is not None, 'can only join a started process'
AssertionError: can only join a started process

Will greatly appreciated any help.
@borgstrom borgstrom self-assigned this Nov 26, 2018
@WisdomWolf
Copy link

I'm seeing a similar issue. I thought it may have been due to PyCharm debugger issues or the fact that I'm running Python 3.8, but I get the same issue outside of PyCharm and this issue shows that someone was having a similar problem way before 3.8 was released.

Traceback (most recent call last):
  File "/Users/ryanbeaman/PycharmProjects/inpatient-realtime-processor/local_consumer/kinesis_consumer.py", line 32, in <module>
    for message in consumer:
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/kinesis/consumer.py", line 188, in __iter__
    self.setup_shards()
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/kinesis/consumer.py", line 150, in setup_shards
    self.shards[shard_data['ShardId']] = ShardReader(
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/kinesis/consumer.py", line 31, in __init__
    self.start()
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 55, in start
    self.process.start()
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 75, in atexit
    recursively_shutdown(cls)
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 71, in recursively_shutdown
    recursively_shutdown(klass)
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 74, in recursively_shutdown
    obj.shutdown()
  File "/Users/ryanbeaman/.pyenv/versions/inpatient-realtime-processor/lib/python3.8/site-packages/offspring/process.py", line 84, in shutdown
    self.process.terminate()
  File "/Users/ryanbeaman/.pyenv/versions/3.8.1/lib/python3.8/multiprocessing/process.py", line 133, in terminate
    self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'

Process finished with exit code 1

@namioto
Copy link

namioto commented Apr 1, 2020

I'm also saw this issue. My python version is 3.8.1.

[INFO] 2020-04-01 16:59:10,173 (kinesis.consumer) Shard reader for shardId-000000000000 does not exist, creating...
[INFO] 2020-04-01 16:59:10,173 (kinesis.consumer) shardId-000000000000 iterator arguments: {'ShardIteratorType': 'LATEST'}
[ERROR] 2020-04-01 16:59:10,207 (main) Can't pickle local object 'Subprocess.start.<locals>.bootstrap'
Traceback (most recent call last):
  File "stream_engine_v2.py", line 105, in <module>
    for message in consumer:
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/kinesis/consumer.py", line 188, in __iter__
    self.setup_shards()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/kinesis/consumer.py", line 150, in setup_shards
    self.shards[shard_data['ShardId']] = ShardReader(
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/kinesis/consumer.py", line 31, in __init__
    self.start()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 55, in start
    self.process.start()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Subprocess.start.<locals>.bootstrap'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 75, in atexit
    recursively_shutdown(cls)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 71, in recursively_shutdown
    recursively_shutdown(klass)
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 74, in recursively_shutdown
    obj.shutdown()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/site-packages/offspring/process.py", line 84, in shutdown
    self.process.terminate()
  File "/opt/miniconda3/envs/control-tower/lib/python3.8/multiprocessing/process.py", line 133, in terminate
    self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'

However, it worked normally in Python 3.7.7.

@corbanvilla
Copy link

For me it was an issue on macOS but not on linux...

It works fine on Python 3.7.7 on macOS but not 3.8.5, though it does work on ubuntu on 3.8.5

@mikenac
Copy link

mikenac commented Oct 26, 2020 •
edited
Loading

Confirmed failure on Mac Python 3.8. It sounds like this might describe the issue: https://chrissardegna.com/blog/multiprocessing-changes-python-3-8/

Cross posted here: borgstrom/offspring#4

@mikenac mikenac mentioned this issue Oct 27, 2020
Open
@mikenac
Copy link

mikenac commented Oct 27, 2020

Left solution over at offspring project. If merged, then it's just a matter of re-pinning the offspring lib at the new version.

@Bhavikpatel576
Copy link

Any updates on this error?

@abiudrodas
Copy link

a workaround is provided in borgstrom/offspring#4, basically you need to add the following line of code to your script

multiprocessing.set_start_method("fork")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@borgstrom borgstrom

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@mikenac @borgstrom @WisdomWolf @corbanvilla @abiudrodas @Bhavikpatel576 @namioto @houdinisparks