Russel Winder's Website

Python Adapts to the Multicore Era

Thanks to Sarah Mount for pointing out Guido van Rossum's summary of his immediate impressions on EuroPython 2010 on getting home and getting rested. Paragraph 2 is clearly the one that I am most interested in as I get a "name check". :-)

Guido's paragraph acts as an excellent executive summary of the message that I was trying to get across. Applications programming in the world of clusters of multicore machines should not be about shared-memory multithreading but should be about small, lightweight processes sending messages to each other. Dare I suggest that this is a return to the true spirit of object-orientation of the 1980s? cf. Smalltalk-80 and all the research languages that were under investigation in the late 1980s (*).

In a world where applications are developed as small interacting processes either CSP-style or actor-style, or even dataflow-style (a model that didn't get much air time at EuroPython 2010 but is arguably equally important) CPython's GIL (global interpreter lock) is much less of an issue than ensuring that the multiprocessing package (and, perhaps bizarrely, the threading package) are as efficient as it is possible for man and machine to make it: multiprocessing (or something basically the same) is the foundation on which actor, dataflow and CSP based frameworks will be based, so it needs to be as fast as it is possible to make it. I have no doubt that the Python- CSP and PyCSP teams (or a merged team if that happens) will pick up on Guido's invitation to make proposals for evolving multiprocessing if it is seen to be needed.

This raises the question of whether there are efforts to create Python packages to present actor-based and dataflow-based models. I haven't seen anything relating to dataflow, and all the actor frameworks I have seen have been solely about handling concurrency in a single PVM context, they have not been about parallelism in a multi-PVM context. So clearly there are two obvious projects here:

  1. Evolve one of the present actor model packages to foster parallelism as well as concurrency.
  2. Start a dataflow model package.

The only questions are who, where and how. Feel free to email me ideas. Hopefully out of this we can create dynamic and productive activities that will see Python programming evolve to meet head on the challenges that multicore computing brings.

Looking over at the JVM-based milieu, the GPars project (**) has brought actors, CSP and dataflow to the Groovy and Java world. Scala, Scalaz and Akka are bringing actors and data parallelism to the JVM. Clojure, with its STM, actors and agents has truly modernized Lisp. The energy and progress in these communities needs to be replicated in the Python community in order for Python to retain relevance in a world increasingly awash with processing cores. So Pythonistas, sign up, your language needs you.

_ (*) Including Solve the language for which I led the design team. Solve was an active object based, parallel, object oriented language with static strong typing and transactional state that we developed as part of a large ESPRIT project 1986-1991. There is no point in reviving this work per se, but I must find and publish all the documents so they are available. _

_ (**) Yes, I am involved in GPars activity. _

Copyright © 2017 Russel Winder -