How to obtain a speedup using Python multiprocessing with cx_Oracle?

471
January 09, 2018, at 4:15 PM

From How to use threading in Python? I took this code sample:

from multiprocessing.dummy import Pool as ThreadPool 
pool = ThreadPool(4) 
results = pool.map(my_function, my_array)

And it is perfectly working with such a urllib.request.urlopen function and gives about 2-3 times speed increase

from urllib.request import urlopen
from multiprocessing.dummy import Pool as ThreadPool
from timeit import Timer
urls = [
  'http://www.python.org', 
  'http://www.python.org/about/',
  'http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html',
  'http://www.python.org/doc/',
  'http://www.python.org/download/',
  'http://www.python.org/getit/',
  'http://www.python.org/community/',
]
def mult(urls):
    pool = ThreadPool(8) 
    results = pool.map(urlopen, urls)
    pool.close()
    pool.join()
def single(urls):
    [urlopen(url) for url in urls]
print(Timer(lambda: single(urls)).timeit(number=1))
print(Timer(lambda: mult(urls)).timeit(number=1))

But in case of calling DB-procedures I did not notice any speedup provided by multiprocessing

from multiprocessing import Pool
import cx_Oracle as ora
import configparser
config = configparser.ConfigParser()
config.read('configuration.ini')
conf = config['sample_config']
dsn = ora.makedsn(conf['ip'], conf['port'], sid=conf['sid'])
connection = ora.Connection(user=conf['user'], password=conf['password'], dsn=dsn, threaded=True)
cursor = ora.Cursor(connection)
def test_function(params):
    cursor = p.connection.cursor()
    # call procedure
    cursor.callproc('Sample_PKG.my_procedure', keywordParameters=params)
dicts = [{'a': 'b'}, {'a': 'c'}] # a lot of dictionaries contains about 30 items each
pool = ThreadPool(4)
pool.map(test_function, dicts)
pool.close()
pool.join()

So, why it so and what could be the solution to boost the script's work?

Rent Charter Buses Company
READ ALSO
How to use Node-mailer without using service option?

How to use Node-mailer without using service option?

Is there any way to use Node-mailer without using service option by using only username and password

210
There was no raw user ticket in the requestPropertyMappings

There was no raw user ticket in the requestPropertyMappings

Iam trying to renew a microsoft webhooks subscriptionBut I get this error:

277
Requests handling inside Event Pool using NodeJS

Requests handling inside Event Pool using NodeJS

I have read the difference between Multi thread mechanism and NodeJS Single thread mechanism hereI know less about thread concept

236