python - Using joblib makes the program run much slower, why? -

python - Using joblib makes the program run much slower, why? -

i have many many little tasks in for loop. want utilize concurrency speed up. used joblib easy integrate. however, found using joblib makes programme run much slower simple for iteration. here demo code:

import time import random os import path import tempfile import numpy np import gc joblib import parallel, delayed, load, dump def func(a, i): '''a simple task demonstration''' a[i] = random.random() def memmap(a): '''use memory mapping prevent memory allocation each worker''' tmp_dir = tempfile.mkdtemp() mmap_fn = path.join(tmp_dir, 'a.mmap') print 'mmap file:', mmap_fn _ = dump(a, mmap_fn) # dump a_mmap = load(mmap_fn, 'r+') # load del gc.collect() homecoming a_mmap if __name__ == '__main__': n = 10000 = np.zeros(n) # memory mapping = memmap(a) # parfor t0 = time.time() parallel(n_jobs=4)(delayed(func)(a, i) in xrange(n)) t1 = time.time()-t0 # t0 = time.time() [func(a, i) in xrange(n)] t2 = time.time()-t0 # joblib time vs time print t1, t2

on laptop i5-2520m cpu, 4 cores, win7 64bit, running time 6.464s joblib , 0.004s simplely for loop.

i've made arguments memory mapping prevent overhead of reallocation each worker. i've reddish relative post, still not solved problem. why happen? did missed disciplines correctly utilize joblib?

"many little tasks" not fit joblib. coarser task granularity, less overhead joblib causes , more benefit have it. tiny tasks, cost of setting worker processes , communicating info them outweigh whatever benefit parallelization.

python performance parallel-processing multiprocessing