I have a python code that uses the subprocess package to run in shell:
subprocess.call(mycode.py, shell=inshell)
When I execute the top command I see that I am only using ~30% or less of CPU. I realize some commands may be using disk and not cpu therefore I was timing the speed. The speed running this on a linux system seems slower than a mac 2 core system.
How do I parallelize this with threading or multiprocessing package so that I can use multiple CPU cores on said linux system?
Well, you can create first a thread, then pass to it the function you want to parallelize. Inside the function you have the subprocess.
import threading
import subprocess
def worker():
"""thread worker function"""
print 'Worker'
subprocess.call(mycode.py, shell=inshell)
return
threads = []
for i in range(5):
t = threading.Thread(target=worker)
threads.append(t)
t.start()
To parallelize the work done in mycode.py
, you need to organize the code so that it fits into this basic pattern:
# Import the kind of pool you want to use (processes or threads).
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool
# Collect work items as an iterable of single values (eg tuples,
# dicts, or objects). If you can't hold all items in memory,
# define a function that yields work items instead.
work_items = [
(1, 'A', True),
(2, 'X', False),
...
]
# Define a callable to do the work. It should take one work item.
def worker(tup):
# Do the work.
...
# Return any results.
...
# Create a ThreadPool (or a process Pool) of desired size.
# What size? Experiment. Slowly increase until it stops helping.
pool = ThreadPool(4)
# Do work and collect results.
# Or use pool.imap() or pool.imap_unordered().
work_results = pool.map(worker, work_items)
# Wrap up.
pool.close()
pool.join()
---------------------
# Or, in Python 3.3+ you can do it like this, skipping the wrap-up code.
with ThreadPool(4) as pool:
work_results = pool.map(worker, work_items)
Firebase Cloud Functions: PubSub, "res.on is not a function"
TypeError: Cannot read properties of undefined (reading 'createMessageComponentCollector')
I am trying to fit a natural log curve to my 2d data in pythonI fit a curve and graphed it and all, the only problem is that matplotlib just connects the dots, but doesn't "curve" it
For example is the following allowed? If so, is it not recommended or is it okay?
I was wondering, How would I go about filtering out people who own over 20 games, and they don't have a Mac OS SystemI need it to be done via a python script, and when run, it outputs its data in a seperate file, like a text file or something
How can I find packet error rate, Channel efficiency and collisions using scapy and python in wifi trafficI have to do this by data from pcap file