Module threadpool
[frames] | no frames]

Module threadpool

source code

Easy to use object-oriented thread pool framework.

A thread pool is an object that maintains a pool of worker threads to perform time consuming operations in parallel. It assigns jobs to the threads by putting them in a work request queue, where they are picked up by the next available thread. This then performs the requested operation in the background and puts the results in another queue.

The thread pool object can then collect the results from all threads from this queue as soon as they become available or after all threads have finished their work. It's also possible, to define callbacks to handle each result as it comes in.

The basic concept and some code was taken from the book "Python in a Nutshell, 2nd edition" by Alex Martelli, O'Reilly 2006, ISBN 0-596-10046-9, from section 14.5 "Threaded Program Architecture". I wrapped the main program logic in the ThreadPool class, added the WorkRequest class and the callback system and tweaked the code here and there. Kudos also to Florent Aide for the exception handling mechanism.

Basic usage:

>>> pool = ThreadPool(poolsize)
>>> requests = makeRequests(some_callable, list_of_args, callback)
>>> [pool.putRequest(req) for req in requests]
>>> pool.wait()

See the end of the module code for a brief, annotated usage example.

Website : http://chrisarndt.de/projects/threadpool/


Version: 1.3.2

Author: Christopher Arndt

License: MIT license

Classes
  NoResultsPending
All work requests have been processed.
  NoWorkersAvailable
No worker threads available to process remaining requests.
  WorkerThread
Background thread connected to the requests/results queues.
  WorkRequest
A request to execute a callable for putting in the request queue later.
  ThreadPool
A thread pool, distributing work requests and collecting results.
Functions
 
makeRequests(callable_, args_list, callback=None, exc_callback=<function _handle_thread_exception at 0x7f4c990347d0>)
Create several work requests for same callable with different arguments.
source code
Function Details

makeRequests(callable_, args_list, callback=None, exc_callback=<function _handle_thread_exception at 0x7f4c990347d0>)

source code 

Create several work requests for same callable with different arguments.

Convenience function for creating several work requests for the same callable where each invocation of the callable receives different values for its arguments.

args_list contains the parameters for each invocation of callable. Each item in args_list should be either a 2-item tuple of the list of positional arguments and a dictionary of keyword arguments or a single, non-tuple argument.

See docstring for WorkRequest for info on callback and exc_callback.