How to use threads to retrieve sub-array

363
December 23, 2016, at 3:31 PM

Assume that I have a numpy array with size of (10,3,4). Now I want to do some operations at subarray among first dimension like the following code:

a = np.arange(120).reshape([10,3,4])
for batch in range(10):
    result = do_something_func( a[batch,:,:] )
    # other operations in for
# other operations

My question is I want to use multi-threads to finish this task without using index to iterate and retrieve the subarray by a for loop. Does anyone know how to do that?

Answer 1

Possibly with gevent:

import gevent
from gevent.pool import Pool
import numpy as np
pool = Pool(8)
def map_subarray(sub_array):
    return sub_array
a = np.arange(120).reshape([10,3,4])
for batch in range(10):
    result = pool.map(map_subarray, a[batch,:,:])
    print(batch, list(result))

Output:

(0, [array([0, 1, 2, 3]), array([4, 5, 6, 7]), array([ 8,  9, 10, 11])])
(1, [array([12, 13, 14, 15]), array([16, 17, 18, 19]), array([20, 21, 22, 23])])
(2, [array([24, 25, 26, 27]), array([28, 29, 30, 31]), array([32, 33, 34, 35])])
(3, [array([36, 37, 38, 39]), array([40, 41, 42, 43]), array([44, 45, 46, 47])])
(4, [array([48, 49, 50, 51]), array([52, 53, 54, 55]), array([56, 57, 58, 59])])
(5, [array([60, 61, 62, 63]), array([64, 65, 66, 67]), array([68, 69, 70, 71])])
(6, [array([72, 73, 74, 75]), array([76, 77, 78, 79]), array([80, 81, 82, 83])])
(7, [array([84, 85, 86, 87]), array([88, 89, 90, 91]), array([92, 93, 94, 95])])
(8, [array([96, 97, 98, 99]), array([100, 101, 102, 103]), array([104, 105, 106, 107])])
(9, [array([108, 109, 110, 111]), array([112, 113, 114, 115]), array([116, 117, 118, 119])])

Reference:

  • http://sdiehl.github.io/gevent-tutorial/
Rent Charter Buses Company
READ ALSO
Scroll down pages from top to bottom

Scroll down pages from top to bottom

I have multiple URLs which is open by python scriptI have to scroll down all the web pages from top to bottom

425
NaN values in the groupby statistics in pandas

NaN values in the groupby statistics in pandas

I am working with a pandas DataFrame Top15 that contains a population data of 15 countries in the world

1107
Stacked Bar Chart with Centered Labels

Stacked Bar Chart with Centered Labels

I'm trying to "robustly" center the data labels in a stacked bar chartA simple code and the result are given below

437
will database pull whole table at once or one row by one row after using model.objects.all().iterator() in django?

will database pull whole table at once or one row by one row after using model.objects.all().iterator() in django?

I know django will return the model object one by one after using iterator() on queryset to save memoryIn database side, will django pull the data one row by one row or still pull the whole table at once just like model

341