Programming languages: Python is slow, but it's about to get faster
Python is incredibly popular because it's easy to learn, versatile, and has thousands of useful libraries for data science. But one thing it is not is fast.
That's about to change in Python 3.11, currently in the first beta phase of its preview (version 3.11.0b1) ahead of its stable release later this year. Core Python (CPython) developer Mark Shannon shared details about the project to make Python faster at the PyCon 2022 conference this week, where developers also showed off progress on the goal of running Python code in the browser.
Developer
Last year, Microsoft funded a project for the Python Software Foundation (PSF), led by Python creator Guido van Rossum and Shannon, to make Python twice as fast as the current stable 3.10 series. The vision is to nudge Python towards the performance of C.
SEE: How to get promoted: Five ways to climb the ladder and have a successful career
Microsoft hired van Rossum in 2020 and gave him a free hand to pick any project. At last year's PyCon 2021 conference, he said he "chose to go back to my roots" and would work on Python's famed lack of performance.
Performance, perhaps, hasn't been a top priority for Python as adoption has been fueled by machine learning and data science thanks to Tensor Flow, Numpy, Pandas and many more platforms, such as AWS's Boto3 SDK for Python. These platforms are downloaded tens of millions of times a month and used in environments that are often not constrained by hardware.
The Faster CPython Project provided some updates about CPython 3.11 performance over the past year. Ahead of PyCon 2022, the project published more results comparing the 3.11 beta preview to 3.10 on dozens of performance metrics, showing that 3.11 was overall 1.25 times faster than 3.10.
Shannon is realistic about the project's ability to improve Python performance, but believes the improvements can extend Python's viable use to more virtual machines.
"Python is widely acknowledged as slow. Whilst Python will never attain the performance of low-level languages like C, Fortran, or even Java, we would like it to be competitive with fast implementations of scripting languages, like V8 for Javascript or luajit for lua," he wrote last year in the Python Enhancement Proposal (PEP) 659.
"Specifically, we want to achieve these performance goals with CPython to benefit all users of Python including those unable to use PyPy or other alternative virtual machines."
The key approach detailed in PEP 659 is a "specializing, adaptive interpreter that specializes code aggressively, but over a very small region, and is able to adjust to mis-specialization rapidly and at low cost."
As noted, optimizations for VMs are "expensive", often requiring a long "warm up" time. To avoid this time expense, the VM should "speculate that specialization is justified even after a few executions of a function", so the the interpreter needs to optimize and de-optimize continually and very cheaply.
This should result in a faster interpreter for CPython that tracks individual bytecodes during a program's execution. The work on the new interpreter is almost complete but still requires completion of dynamic specializations for loops and binary operations, according to PSF.
Additionally, memory consumption in 3.11 hasn't changed from 3.10.
SEE: Developers are facing burnout. Here's how companies are trying to fix it
On the question of a just-in-time (JIT) compiler for Python's performance, Shannon suggested it was not a priority and would likely not arrive until Python 3.13, according to the Python Software Foundation's coverage of the event.
Anaconda, the maker of the Anaconda Python distribution for data science, is backing the Pyston project, an implementation of Python that promises speed improvements over Python.
One of Anaconda's older efforts to speed up Python was the Numba project, an LLVM-based JIT compiler for CPython, which accelerates numerical Python functions running on the CPU or GPU, but can't optimize entire programs and doesn't address wider Python use cases. Another is PyPy, an implementation of CPython with a JIT compiler for faster performance.
According to the Faster Python implementation plan, CPython 3.12 might gain a "simple JIT compiler for small regions" that compiles small regions of specialized code, while 3.13 would enhance the compiler to extend the regions for compilation.