Python 3.12 Preview: Support For the Linux perf Profiler

What’s the Linux perf Profiler?

If you search online for information about the Linux perf profiler, then you may get confused by the plethora of names that people use to talk about it. To make matters worse, the profiler seems to be poorly documented. You really need to dig deep, as the corresponding article on Wikipedia and the official Wiki page don’t provide a lot of help.

The Linux perf profiler actually consists of two high-level components:

1. perf_events, or performance counters for Linux (PCL): A subsystem in the Linux kernel with an API that provides an abstraction layer over a hardware-specific performance monitoring unit (PMU) present in modern architectures. The PMU consists of special CPU registers or counters for collecting metrics about events such as the number of cache misses, CPU cycles, or executed instructions. This information is invaluable during performance analysis.

2. perf command-line tool: A user-space utility tool built on top of the perf_events API from the Linux kernel, which can help you collect and make sense of the data. It follows the same philosophy as Git by offering several specialized subcommands. You can list them all by typing perf at your command prompt.

The first component is baked right into the Linux kernel, meaning that recent versions of mainstream Linux distributions will usually have it shipped and enabled. On the other hand, you’ll most likely need to install an additional package to start using the perf command, as it’s not essential for regular users.

While the Linux perf profiler came into existence primarily as an interface to hardware-level performance counters, it also covers numerous software performance counters. For example, you can use it to track the number of context switches made by your operating system’s task scheduler or the number of times a running thread has migrated from one CPU core to another.

Okay, but are there any other benefits of using the Linux perf profiler aside from its hardware capabilities?

How Can Python Benefit From perf?

For several reasons, adding support for the Linux perf profiler will greatly impact your ability to profile Python applications. There’s no better source to refer to than Pablo Galindo Salgado, who’s the primary contributor behind this new feature. In a Twitter announcement, he thoroughly explains the benefits of the Python and perf integration:

Python 3.12 will add support for the Linux perf profiler! 🔥🔥 Perf is one of the most powerful and performant profilers for Linux that allows getting a ridiculous amount of information such as CPU counters, cache misses, context switching, and much more. (Source)

The lengthy tweet thread gives examples of several new capabilities:

• Python function call recognition: The interpreter’s support will allow perf to recognize Python function calls in the call stack, so the profile report will show Python code alongside code written in languages like C++ or Rust.

• Isolation of performance bottlenecks: Many Python libraries, such as NumPy and pandas, call functions in foreign languages. Now, you’ll be able to identify whether your Python code is the actual performance bottleneck, which will help you decide whether to improve your Python code or change how you use external libraries.

• Access to low-level system and hardware events: This will enable you to find out, for example, how much Python’s global interprer lock (GIL) is affecting your code’s performance.

• Vast ecosystem of perf tools: This includes performance visualization tools, which you’ll explore later in this tutorial.

In short, perf will give you holistic insight into the performance of the entire stack, from hardware and the operating system to the user-level code, while inflicting minimal overhead. That’s an incredibly powerful development for you as a Python developer, allowing you to quickly find and address critical bottlenecks that affect the performance of your applications.

Additionally, the Linux perf profiler will bring the following benefits into the Python world:

• Much less overhead: Compared to other profilers, it’ll have a noticeably smaller memory and CPU footprint, letting you squeeze a lot more out of your profiled programs.

• Exposure of hardware events: Access to events like CPU cache misses can sometimes be invaluable in optimizing low-level code interactions with your hardware.

• View of the kernel-level code: The ability to profile at the system level will let you identify performance bottlenecks outside the Python code, such as input and output operations, system calls, or third-party libraries used within your program.

• Easier concurrency: Profiling multithreaded or multiprocess Python applications, including those that run on more than one CPU core at a time, will become easier.

• Better integration with existing tools: You’ll gain access to a rich ecosystem of tools for profiling and optimizing performance across different languages and environments. This can lead to a more consistent workflow and a reduced learning curve.

All that sounds great. Unfortunately, things get tricky when you try to profile an interpreted language like Python because perf was designed to work with compiled languages. In the next section, you’ll find out exactly how to make perf work with your Python code.

Install the Linux Tools Package

If you’re on a Debian-based distribution, such as Ubuntu, then open your terminal with Ctrl+Alt+T and update your package information using APT:

$ sudo apt update

After you provide your administrator password, this command will update your local package index by downloading the latest metadata about the packages available in remote sources. Afterward, you’ll be ready to install the most recent version of perf, which is bundled with a few other utility tools in a virtual package called linux-tools on distributions derived from Debian.

Canonical, the company behind Ubuntu, provides a few flavors of the Linux kernel optimized for different scenarios, such as a client computer, a server, an IoT device, or the cloud. If you’re running Ubuntu Desktop, then your Linux kernel is probably called generic, which you can verify by running the following command:

$ cat /proc/version_signature
Ubuntu 5.19.0-43.44-generic 5.19.17

In this case, the mainline kernel version is 5.19.17, and the kernel flavor is generic. That’s important because you must install a specific variant of the linux-tools package tailored to your Linux kernel, like so:

$ sudo apt install linux-tools-generic

Installing the linux-tools-generic package will automatically pull dependencies suitable to your system at that moment. For example, it may download and install the linux-tools-5.19.0-43 package with the perf executable compiled against that particular kernel release, bringing additional features, bug fixes, and optimizations.

In the future, you may need to reinstall the corresponding package—for example, after a system update automatically upgrades your Linux kernel. In such a case, the perf command will likely stop working. When this happens, you can find your current kernel release with the uname -r command and append its output to the package name:

$ uname -r
5.19.0-43-generic

$ sudo apt install linux-tools-`uname -r`

You can do so manually, or you can enclose the command uname -r in backticks for the shell to evaluate it. For instance, the command above would install the linux-tools-5.19.0-43-generic package, which depends on linux-tools-5.19.0-43.

After the installation is complete, you’ll have the perf tool available at your fingertips. You can verify this by running the following executable:

$ perf

 usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]

 The most commonly used perf commands are:
   (...)

 See 'perf help COMMAND' for more information on a specific command.

Running perf without any arguments will list the available subcommands. Remember that you may need to run some of them with sudo for superuser access, which is required for accessing kernel-level events and function calls. In some cases, you can use a suitable command-line option to filter out those privileged resources.

The remaining ingredient for using perf with Python is the Python 3.12 interpreter, which you’ll install now.

Install Python 3.12 Using pyenv

Unless you already have Python 3.12 or newer installed on your computer, you can quickly and painlessly get the desired version with pyenv. At the time of writing, the third beta preview of Python 3.12 was the latest release of Python:

$ pyenv install --list | grep 3.12
  3.12.0b3
  3.12-dev

You’ll soon use the version specifier, such as 3.12.0b3, to download and install the corresponding Python interpreter with pyenv. However, you need to specify some build flags first.

For best results with perf, you should make sure to build your Python interpreter using a set of extra compiler flags (CFLAGS), which disable certain optimizations. Otherwise, perf might not be able to capture the entire stack trace, failing to show some of your Python functions in the report. For any Python installation, you can check if it was compiled with those flags by issuing these commands:

$ python -m sysconfig | grep no-omit-frame-pointer
$ python -m sysconfig | grep no-omit-leaf-frame-pointer

If there’s no output, then it means that Python has been compiled without those flags. While you can still get some results with perf, the profiler won’t know about many of your Python function calls. Additionally, it might report those that it does capture as unknown, making it difficult to read the performance profile:

Samples: 65K of event 'cycles', Event count (approx.): 22620504399
  Children      Self  Command      Shared Object      Symbol
-   32.47%     0.00%  python3      [unknown]          [.] 0x00007fa5ae88fea0
   - 0x7fa5ae88fea0
      + 6.98% 0x7ffc2f6a7e40
      + 6.18% 0x7ffc2f6a7bc0
      + 5.06% 0x7ffc2f6a7990
      + 3.47% 0x7ffc2f6a8370
      + 3.23% 0x7ffc2f6a8070
      + 2.23% 0
      + 2.11% 0x7ffc2f6a80f0
      + 1.43% 0x7ffc2f6a7c10
      + 0.90% 0x7fa5ae3252f8
(...)

That doesn’t look very helpful, does it?

In order to get more information into the profile, you need to specify the necessary compiler flags before running pyenv install:

$ export PYTHON_CFLAGS='-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer'
$ pyenv install 3.12.0b3
Downloading Python-3.12.0b3.tar.xz...
-> https://www.python.org/ftp/python/3.12.0/Python-3.12.0b3.tar.xz
Installing Python-3.12.0b3...
Installed Python-3.12.0b3 to /home/realpython/.pyenv/versions/3.12.0b3

The tool downloads Python source code, which means that you must have build dependencies like the C compiler in place to proceed. When the installation is finished, you’ll be able to set the global hash shortcut to python in your shell so that it points to the newly installed interpreter:

$ pyenv global 3.12.0b3
$ python --version
Python 3.12.0b3

This will make the python command run whatever Python version you requested.

You get both convenience and flexibility when installing Python with pyenv. However, there are alternatives. If you have the know-how and a few extra minutes on your hands, then you may compile Python yourself with the necessary flags for perf profiling. You’ll find detailed instructions on how to do it in the next section.

Install Python 3.12 From Source Code

To make a custom Python build, you can clone the desired branch straight from the CPython repository on GitHub. You may experiment with the bleeding edge on the main branch or grab one of the tagged releases, such as this one:

$ git clone --branch v3.12.0b3 https://github.com/python/cpython.git

Because you’re cloning a specific Git tag, you’ll see a warning message afterward telling you that you’re in a so-called detached HEAD state. This means that you’re not currently on a branch, so any potential commits won’t be associated with any branch. However, there’s nothing to worry about because you’re not going to make any changes.

Alternatively, if you don’t have a Git client available in your terminal, then you can browse the repository tags using your web browser and download the corresponding version in the form of an archive file to extract.

Once you navigate to the folder where you’ve downloaded the Python source code, you can compile and install it:

$ cd cpython/
$ export CFLAGS='-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer'
$ ./configure --prefix="$HOME/python-custom-build"
$ make -j $(nproc)
$ make install

You start by setting the CFLAGS environment variable, instructing the compiler to retain debugging information related to frame pointers in the function call stack. That’s the crucial step missing from the default installation using pyenv.

Next, you run the configuration script to generate a Makefile, while specifying the target directory for Python’s installation. In this case, you install your custom Python build into the python-custom-build/ folder in your home directory.

Finally, you compile and install Python using the generated Makefile with make. The compilation step can take a few long minutes. Therefore, it’s common to speed up this process by running a few threads in parallel. The nproc command determines the number of processors available on your computer, and the -j flag tells make to use that many threads.

While the Python interpreter is compiling, you’ll see a lot of cryptic output in your terminal. You might even glimpse a few ominous warnings, which are usually safe to ignore. Only get worried if there’s an actual error preventing the compilation from completing successfully. But, if that happens, then you’ll usually see a helpful message pointing to the root cause of the problem.

At this point, you should be able to run your shiny new Python interpreter by specifying the path to the respective python3 executable:

$ $HOME/python-custom-build/bin/python3
Python 3.12.0b3 (tags/v3.12.0b3:f992a60014, Jun 29 2023, 14:24:02) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

The Python REPL’s output reveals your interpreter version, including the abbreviated Git commit hash that you cloned, your compiler version, and even the date and time of the compilation!

However, typing the whole path to your Python executable each time you want to run it is rather inconvenient. To make your life easier, you could add the corresponding bin/ folder’s path to your $PATH variable or create a temporary alias in your shell as needed:

$ alias custom-python=$HOME/python-custom-build/bin/python3

With that, you’ll be able to type custom-python at your command prompt to run Python 3.12. In particular, you can now check if you indeed built the Python interpreter with the frame pointer optimizations disabled:

$ custom-python -m sysconfig | grep 'no-omit-frame-pointer'
    CFLAGS = "(...) -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
    CONFIGURE_CFLAGS = "-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
    CONFIG_ARGS = "'--prefix=/home/realpython/python-custom-build...
    LIBEXPAT_CFLAGS = "-I./Modules/expat -fno-strict-overflow -DNDEBUG...
    LIBHACL_CFLAGS = "-I./Modules/_hacl/include -D_BSD_SOURCE...
    LIBMPDEC_CFLAGS = "-I./Modules/_decimal/libmpdec -DCONFIG_64=1...
    PY_BUILTIN_MODULE_CFLAGS = "-fno-strict-overflow -DNDEBUG -g -O3...
    PY_CFLAGS = "-fno-strict-overflow -DNDEBUG -g -O3 -Wall...
    PY_CORE_CFLAGS = "-fno-strict-overflow -DNDEBUG -g -O3 -Wall...
    PY_STDMODULE_CFLAGS = "-fno-strict-overflow -DNDEBUG -g -O3 -Wall...

Excellent! Both the -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer flags appear in CFLAGS and a few related variables. This means you can start using perf with Python without restrictions!

Now that you have perf and Python 3.12 installed, you must learn how to enable a special mode in which to run the interpreter. Otherwise, your Python functions will remain invisible to perf, which can only understand native code compiled from languages like C, C++, Go, and Rust. You’ll fix that now!

Run Python With Trampoline Mode Enabled

If you now profiled a script using perf and Python 3.12 without taking extra steps, then you wouldn’t see any of your functions in the report. Instead, the deepest frame in the call stack exhibiting a significant cost would show a native wrapper function from the CPython interpreter:

Samples: 10K of event 'cycles', Event count (approx.): 34386234823

-  100.00%     10680
   -   99.77%     /home/realpython/python-custom-build/bin/python3.12
        __libc_start_main
      - Py_BytesMain
         - 99.89% pymain_run_python.constprop.0
              _PyRun_AnyFileObject
              _PyRun_SimpleFileObject
              run_mod
              run_eval_code_obj
            - PyEval_EvalCode
               + 92.95% _PyEval_EvalFrameDefault
                 2.27% _PyObject_Free
                 2.18% long_dealloc
                 1.34% PyObject_Free
                 0.80% PyLong_FromLong
   +    0.23%     /proc/kcore

That wrapper function, _PyEval_EvalFrameDefault(), is implemented in C and is responsible for evaluating the underlying Python bytecode at runtime. Unfortunately, it renders the profiler’s report unhelpful because the same C function evaluates all the Python functions. So, you won’t know exactly which Python function is taking the longest to execute.

To associate each call to that evaluation function with a corresponding Python counterpart, you must enable trampoline instrumentation, which is disabled by default. In trampoline mode, the Python interpreter will insert a dynamically generated trampoline frame into the native call stack, acting as glue code between C and Python. This, in turn, will allow for displaying non-native function names on the call stack:

Samples: 10K of event 'cycles', Event count (approx.): 34217160648

-  100.00%     10432
   -   99.16%     /home/realpython/python-custom-build/bin/python3.12
        __libc_start_main
      - Py_BytesMain
         - 99.89% pymain_run_python.constprop.0
              _PyRun_AnyFileObject
              _PyRun_SimpleFileObject
              run_mod
              run_eval_code_obj
              PyEval_EvalCode
              py::<module>:/home/realpython/project/script.py
              _PyEval_EvalFrameDefault
              PyObject_Vectorcall
              py::main:/home/realpython/project/script.py
              _PyEval_EvalFrameDefault
            - PyObject_Vectorcall
               + 66.61% py::slow_function:/home/realpython/project/script.py
               + 33.39% py::fast_function:/home/realpython/project/script.py
   +    0.84%     /proc/kcore

Now, each call to the evaluation function is accompanied by a clearly named trampoline call, which intercepts a Python function just before delegating the execution to _PyEval_EvalFrameDefault(). You can see that Python evaluates a module named script. It then enters the main() function in that module and calls slow_function() and fast_function() in some unknown order. The former takes roughly twice as much time to finish as the latter.

Before moving forward, it’s worthwhile to double-check if your Python interpreter supports perf profiling in the first place. To do so, you can inspect the following configuration variable, assuming you still have that custom-python alias defined in your shell:

$ custom-python -m sysconfig | grep PY_HAVE_PERF_TRAMPOLINE
        PY_HAVE_PERF_TRAMPOLINE = "1"

If not, then you can always provide the full path to the python executable. If you don’t see any output from the command above, or the PY_HAVE_PERF_TRAMPOLINE variable has a value of zero, then it means your Python interpreter won’t be able to enable trampoline mode on your platform. Otherwise, you’re ready to continue.

There are three ways to enable trampoline instrumentation in Python 3.12:

1. Environment variable: Set the PYTHONPERFSUPPORT=1 environment variable.
2. Interpreter option: Use the python -X perf option.
3. Python API: Call sys.activate_stack_trampoline("perf").

The first two ways allow you to enable perf profiling support for the entire Python interpreter. On the other hand, with the last one, you can temporarily enable or disable the trampoline at runtime. This can be useful for isolating bits of code to profile from the rest of your program.

Here are the first two ways of activating the stack trampoline in action:

$ CODE='import sys; print(sys.is_stack_trampoline_active())'

$ custom-python -c "$CODE"
False

$ PYTHONPERFSUPPORT=1 custom-python -c "$CODE"
True

$ custom-python -X perf -c "$CODE"
True

For convenience, you define a one-line Python snippet in a helper environment variable called CODE, which you reference in the subsequent commands. As you can see, Python has the trampoline instrumentation disabled by default because it incurs unnecessary overhead and adds an extra hoop in the call stack.

If you need to turn the trampoline on or off at runtime, potentially more than once, then you can instead use the new Python API in the sys module:

>>> import sys

>>> sys.is_stack_trampoline_active()
False

>>> sys.activate_stack_trampoline("perf")
>>> sys.is_stack_trampoline_active()
True

>>> sys.deactivate_stack_trampoline()
>>> sys.is_stack_trampoline_active()
False

To activate the trampoline using this method, you have to pass a required positional-only parameter with the name of a profiler back end. At the moment, "perf" is the only supported back end.

Okay, now that you have perf installed and Python 3.12 compiled with frame pointer optimization disabled, you can start analyzing complete stack traces in your Python scripts. Next up, you’ll learn how to take advantage of perf to monitor how your Python programs execute.

Prepare a Benchmark Script

The Linux perf profiler lets you scrutinize your application as if you were looking at it under a microscope. To showcase the full potential and advantages of perf over other profiling tools, you’ll import a third-party Python library with a C extension module. As you learned earlier, Pillow provides pre-built Python wheels for Python 3.12, so you’ll use that image processing library as an example in this tutorial.

Go ahead and install Pillow into a virtual environment based on Python 3.12. You can switch to the right interpreter version using pyenv or the shell alias that you defined before:

$ pyenv shell 3.12.0b3
$ python -m venv venv/ --prompt benchmark
$ source venv/bin/activate
(benchmark) $ python -m pip install Pillow

The pyenv shell command makes the specified Python interpreter active in your current shell session. From now on, python will point to that version of Python, as long as you installed it before with pyenv. But, once you open another terminal window, the command will return to its default value.

Remember that pyenv enables compiler optimizations that erase the frame pointer from Python, making the profiler’s report look much less helpful. To create a virtual environment using your custom build of Python, you must specify the full path to the corresponding python3 executable or use a shell alias:

$ alias custom-python=$HOME/python-custom-build/bin/python3
$ custom-python -m venv venv/ --prompt benchmark
$ source venv/bin/activate
(benchmark) $ python -m pip install Pillow

If you don’t have Python 3.12 installed yet, then jump back to the setup instructions in the previous section for more information.

Now, it’s too easy to forget about enabling the trampoline mode in Python 3.12 or to accidentally run the benchmark through the wrong interpreter by specifying an incorrect virtual environment. To save yourself from these troubles, you can write a few safety checks. Create a new Python script named benchmark.py and type the following code in it:

 1# benchmark.py
 2
 3import sys
 4import sysconfig
 5
 6def check_perf_support():
 7    if sys.version_info < (3, 12):
 8        version = sysconfig.get_python_version()
 9        raise RuntimeError(f"This is Python {version}, not 3.12 or later")
10
11    if not sysconfig.get_config_var("PY_HAVE_PERF_TRAMPOLINE"):
12        raise RuntimeError("Python doesn't support perf on this platform")
13
14    if not sys.is_stack_trampoline_active():
15        raise RuntimeError("Did you forget the '-X perf' option?")
16
17    cflags = sysconfig.get_config_var("CONFIGURE_CFLAGS")
18    if "-fno-omit-frame-pointer" not in cflags:
19        print("Python compiled without the frame pointer", file=sys.stderr)
20
21def main():
22    ...
23
24if __name__ == "__main__":
25    try:
26        check_perf_support()
27    ex