python - multiprocessing pool with map -
i try multiprocess code this:
param_seq = [] pool = pool(multiprocessing.cpu_count()-1) def work(param_set): (_, v) = experiment_sim(param_set[0], param_set[1]) v_dev = np.subtract(exp_v, v) return - np.sum(np.square(v_dev)) / (2 * noise_sigma ** 2) x in ra.values: y in gpas.values: param_seq.append((x, y)) likelihood = pool.map(work, param_seq) and looks working fine, work, in , stops , nothing , have stop program. following message occurs:
process poolworker-8: process poolworker-10: process poolworker-13: process poolworker-14: process poolworker-11: process poolworker-12: traceback (most recent call last): traceback (most recent call last): traceback (most recent call last): traceback (most recent call last): traceback (most recent call last): traceback (most recent call last): file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self.run() self.run() self._target(*self._args, **self._kwargs) file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self.run() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self.run() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) self._target(*self._args, **self._kwargs) file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker self._target(*self._args, **self._kwargs) self._target(*self._args, **self._kwargs) file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker self.run() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker self._target(*self._args, **self._kwargs) file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker task = get() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in task = get() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in task = get() task = get() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in task = get() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in task = get() file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in racquire() racquire() racquire() keyboardinterrupt racquire() keyboardinterrupt racquire() racquire() keyboardinterrupt keyboardinterrupt keyboardinterrupt keyboardinterrupt does know can problem?
edit:
finishes computations, can't give result list. program terminated @ point forever.
edit:
solved issue:
# multi processing if __name__ == '__main__': pool = pool(multiprocessing.cpu_count()) likelihood = pool.map(work2, param_seq) pool.close() pool.join() but if use heavy computation, whole thing crashes:
def work2(param_set): (_, v) = experiment_sim(param_set[0], param_set[1]) v_dev = np.array(np.subtract(exp_v, v)) return - 1 / 2 * np.dot(v_dev, invcovmat.dot(v_dev)) where invcovmat huge matrix! there idea how solve problem?
crash report:
process: python [51365] path: /usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/resources/python.app/contents/macos/python identifier: python version: 2.7.11 (2.7.11) code type: x86-64 (native) parent process: python [49361] responsible: python [51365] user id: 502 exception type: exc_bad_access (sigsegv) exception codes: kern_invalid_address @ 0x0000000000000110 exception note: exc_corpse_notify termination signal: segmentation fault: 11 termination reason: namespace signal, code 0xb terminating process: exc handler [0] . . . application specific information: *** multi-threaded process forked *** crashed on child side of fork pre-exec
Comments
Post a Comment