CnC-Python: Multicore Programming with High Productivity

We introduce CnC-Python, an implementation of the Concurrent Collections (CnC) programming model for Python computations. Python has been gaining popularity in multiple domains because of its expressiveness and high productivity. However, exploiting multicore parallelism in Python is comparatively tedious since it requires the use of low-level threads or multiprocessing modules. CnC-Python, being implicitly parallel, avoids the use of these low-level constructs, thereby enabling Python programmers to achieve task, data and pipeline parallelism in a declarative fashion while only being required to describe the program as a coordination graph with serial Python code for individual steps. The CnC-Python runtime requires that Python objects communicated between steps be serializable (picklable), but imposes no restriction on the Python idioms used within the serial code. Programs written in CnC-Python are deterministic in that they produce the same outputs for the same inputs regardless of the execution schedules used by the runtime. Our implementation of CnC-Python uses the CnC Habanero-Java (HJ) runtime system, the Babel compiler to generate glue code while invoking Python from Java, and the multiprocessing module available in standard distributions of Python. However, the CnC-Python user need not be aware about Java, Babel, HJ,or any other component in our runtime to use our system. The HJ-based implementation allows us to reuse a mature CnC runtime for scheduling, and enables us to bypass the well publicized performance issues with the Python interpreter’s global interpreter lock.