Python uncatchable IOError [Errno 32] Broken pipe from Popen within Pool

57
December 02, 2020, at 04:40 AM

I have a Python 2.7 code which is running two piped suprocess Popens with multiprocessing Pool as follows

import sys, shlex
import subprocess as sp
import multiprocessing as mp
try:
        def run(i):
                p0 = sp.Popen(shlex.split('myapp'), stdout=sp.PIPE, stderr=sp.PIPE, shell=False)
                p1 = sp.Popen(shlex.split('cat'), stdin=p0.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
                out, err = p1.communicate()
                return i
        pool = mp.Pool(1)
        pool.map(run, range(1, 100))
except:
        print 'CATCHED'

The issue is that, when myapp is failing with return code 1, an error IOError: [Errno 32] Broken pipe is raised but it does not get caught by the try ... except block which covers everything.

Does anyone know how this is possible? How can I capture this error?

Additional information:

  1. myapp corresponds to bwa mem -M hg19.fa A B where bwa is failing because it does not find a needed index file within current directory where file hg19.fa exists.

  2. I have also noticed that after failing myapp (i.e. bwa) is taking a while to terminate, probably causing the broken pipe as cat has probably already terminated.

  3. The broken pipe seems to occurs when the number of parallel Pool workers is higher than available processors.

  4. Substituting p1.communicate() with p0.wait() and p1.wait() does not change the outcome.

  5. Running with Python 3 (substituting the print statement) seems to be taking ages to terminate but it does not end with the same uncaught error; thus myapp seems to have troubles terminating but Python 3 correctly handles the situation.

READ ALSO
Best way to store user notification prefrences in database

Best way to store user notification prefrences in database

I want to store user notification preferencesI do it like this:

53
MailGun results in 401 forbidden using nodeJs

MailGun results in 401 forbidden using nodeJs

I am trying to send an email using mailgun and nodejsI took the code MailGun provides you and added my domain name and api key:

47
Do something if product type is changed

Do something if product type is changed

I have a custom WooCommerce product typeIf you switched between the standard WooCommerce type the input are cleared but this does not work for custom types

65
Why IntelliJ has inspections for generic usage disabled by default?

Why IntelliJ has inspections for generic usage disabled by default?

Why does IntelliJ has inspections for generic usage disabled by default?

68