python spreading subprocess.call on multiple CPU cores

551
January 13, 2017, at 10:17 AM

I have a python code that uses the subprocess package to run in shell:

subprocess.call(mycode.py, shell=inshell)

When I execute the top command I see that I am only using ~30% or less of CPU. I realize some commands may be using disk and not cpu therefore I was timing the speed. The speed running this on a linux system seems slower than a mac 2 core system.

How do I parallelize this with threading or multiprocessing package so that I can use multiple CPU cores on said linux system?

Answer 1

Well, you can create first a thread, then pass to it the function you want to parallelize. Inside the function you have the subprocess.

import threading
import subprocess
def worker():
    """thread worker function"""
    print 'Worker'
    subprocess.call(mycode.py, shell=inshell)
    return
threads = []
for i in range(5):
    t = threading.Thread(target=worker)
    threads.append(t)
    t.start()
Answer 2

To parallelize the work done in mycode.py, you need to organize the code so that it fits into this basic pattern:

# Import the kind of pool you want to use (processes or threads).
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool
# Collect work items as an iterable of single values (eg tuples, 
# dicts, or objects). If you can't hold all items in memory,
# define a function that yields work items instead.
work_items = [
    (1, 'A', True),
    (2, 'X', False),
    ...
]
# Define a callable to do the work. It should take one work item.
def worker(tup):
    # Do the work.
    ...
    # Return any results.
    ...
# Create a ThreadPool (or a process Pool) of desired size.
# What size? Experiment. Slowly increase until it stops helping.
pool = ThreadPool(4)
# Do work and collect results.
# Or use pool.imap() or pool.imap_unordered().
work_results = pool.map(worker, work_items)
# Wrap up.
pool.close()
pool.join()
---------------------
# Or, in Python 3.3+ you can do it like this, skipping the wrap-up code.
with ThreadPool(4) as pool:
    work_results = pool.map(worker, work_items)
Rent Charter Buses Company
READ ALSO
How do I make this log curve an actual curve?

How do I make this log curve an actual curve?

I am trying to fit a natural log curve to my 2d data in pythonI fit a curve and graphed it and all, the only problem is that matplotlib just connects the dots, but doesn't "curve" it

357
Can you have two of the same regex but point to them to different url files

Can you have two of the same regex but point to them to different url files

For example is the following allowed? If so, is it not recommended or is it okay?

329
How to filter out specific data from a CSV via Python?

How to filter out specific data from a CSV via Python?

I was wondering, How would I go about filtering out people who own over 20 games, and they don't have a Mac OS SystemI need it to be done via a python script, and when run, it outputs its data in a seperate file, like a text file or something

705
How can I calculate PER, collisions and channel efficiency using scapy?

How can I calculate PER, collisions and channel efficiency using scapy?

How can I find packet error rate, Channel efficiency and collisions using scapy and python in wifi trafficI have to do this by data from pcap file

418