_communicate in Popen (subprocess.py) appears to sporadically hang on Apple M1 hardware

[waltstuart25]

Hi, I'm using 3.0.14 on Apple M1 hardware with Python 3.8.10 virtual environment. I'm experiencing sporadic but frequent hangs. When debugging the code, it appears that the hang results from the following while loop in _communicate never exiting:

                while selector.get_map():
                    timeout = self._remaining_time(endtime)
                    if timeout is not None and timeout < 0:
                        self._check_timeout(endtime, orig_timeout,
                                            stdout, stderr,
                                            skip_check_and_raise=True)
                        raise RuntimeError(  # Impossible :)
                            '_check_timeout(..., skip_check_and_raise=True) '
                            'failed to raise TimeoutExpired.')

                    ready = selector.select(timeout)
                    self._check_timeout(endtime, orig_timeout, stdout, stderr)

                    # XXX Rewrite these to use non-blocking I/O on the file
                    # objects; they are no longer using C stdio!

                    for key, events in ready:
                        if key.fileobj is self.stdin:
                            chunk = input_view[self._input_offset :
                                               self._input_offset + _PIPE_BUF]
                            try:
                                self._input_offset += os.write(key.fd, chunk)
                            except BrokenPipeError:
                                selector.unregister(key.fileobj)
                                key.fileobj.close()
                            else:
                                if self._input_offset >= len(self._input):
                                    selector.unregister(key.fileobj)
                                    key.fileobj.close()
                        elif key.fileobj in (self.stdout, self.stderr):
                            data = os.read(key.fd, 32768)
                            if not data:
                                selector.unregister(key.fileobj)
                                key.fileobj.close()
                            self._fileobj2output[key.fileobj].append(data)

            self.wait(timeout=self._remaining_time(endtime))

I can repeat this on many of the functions in the test code. Any ideas on this or whether pyboolnet 3.0.14 has been tested on M1 hardware?

[hklarner]

• What Pyboolnet function are you calling? Might have to do something with binaries which you could call directly and see if you can reproduce the hanging.
• What about your input? Does it really sometimes hang and sometimes not for identical input?

[waltstuart25]

The function test_completness in the test code test_attractors.py (tests folder) always seems to hang in this way. It appears the aforementioned while loop is entered via the first call to bnet2primes(). Further investigation indicates that the hang appears to be in line 415 of the select() function in the _PollLikeSelector class which is in selectors.py:

fd_event_list = self._selector.poll(timeout)

For now, I have switched to a Linux machine and all is working. However, for those running Mac M1 it would be nice to see if we can fix this.

[janvasiljevic]

Hey, I have the same problem. I think it's a problem with the included binaries. By installing clingo (brew install clingo) and then changing the config in dist-packages/pyboolnet/binaries/settings.cfg from:

gringo          = ./gringo-4.4.0/gringo_mac64
clasp           = ./clasp-3.2.0/clasp-3.2.0_mac64

to

gringo          = /opt/homebrew/bin/gringo 
clasp           = /opt/homebrew/bin/clasp 

it started to work.

[hklarner]

@janvasiljevic yes, might be that the shipped mac binaries are outdated.