Serialize Your Data With Python

Textual vs Binary

At the end of the day, all serialized data becomes a stream of bytes regardless of its original shape or form. But some byte values—or their specific arrangement—may correspond to Unicode code points with a meaningful and human-readable representation. Data serialization formats whose syntax consists purely of characters visible to the naked eye are called textual data formats, as opposed to binary data formats meant for machines to read.

The main benefit of a textual data format is that people like you can read serialized messages, make sense of them, and even edit them by hand when needed. In many cases, these data formats are self-explanatory, with descriptive element or attribute names. For example, take a look at this excerpt from the Real Python web feed with information about the latest tutorials and courses published:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Real Python</title>
  <link href="https://realpython.com/atom.xml" rel="self"/>
  <link href="https://realpython.com/"/>
  <updated>2023-09-15T12:00:00+00:00</updated>
  <id>https://realpython.com/</id>
  <author>
    <name>Real Python</name>
  </author>
  <entry>
    <title>Bypassing the GIL for Parallel Processing in Python</title>
    <id>https://realpython.com/python-parallel-processing/</id>
    <link href="https://realpython.com/python-parallel-processing/"/>
    <updated>2023-09-13T14:00:00+00:00</updated>
    <summary>In this tutorial, you&#x27;ll take a deep dive (...)</summary>
    <content type="html">
      &lt;div&gt;&lt;p&gt;Unlocking Python's true potential (...)
    </content>
  </entry>
  (...)
</feed>

The Real Python feed uses the XML-based Atom Syndication Format, which is a form of a data serialization format. You can open the document above in any text editor without needing specialized software or libraries. Furthermore, just by looking at this feed, you can immediately break down its structure and probably guess the meaning of the individual elements without checking the format’s specification.

Now, compare the text-based Atom feed above with an equivalent binary feed. You can generate one by parsing atom.xml with xmltodict and dumping the resulting dictionary to a file using MongoDB’s BSON format. When you do, the result will look something like this:

$ hexdump -C atom.bson | head
00000000  51 d2 01 00 03 66 65 65  64 00 46 d2 01 00 02 40  |Q....feed.F....@|
00000010  78 6d 6c 6e 73 00 1c 00  00 00 68 74 74 70 3a 2f  |xmlns.....http:/|
00000020  2f 77 77 77 2e 77 33 2e  6f 72 67 2f 32 30 30 35  |/www.w3.org/2005|
00000030  2f 41 74 6f 6d 00 02 74  69 74 6c 65 00 0c 00 00  |/Atom..title....|
00000040  00 52 65 61 6c 20 50 79  74 68 6f 6e 00 04 6c 69  |.Real Python..li|
00000050  6e 6b 00 72 00 00 00 03  30 00 3f 00 00 00 02 40  |nk.r....0.?....@|
00000060  68 72 65 66 00 20 00 00  00 68 74 74 70 73 3a 2f  |href. ...https:/|
00000070  2f 72 65 61 6c 70 79 74  68 6f 6e 2e 63 6f 6d 2f  |/realpython.com/|
00000080  61 74 6f 6d 2e 78 6d 6c  00 02 40 72 65 6c 00 05  |atom.xml..@rel..|
00000090  00 00 00 73 65 6c 66 00  00 03 31 00 28 00 00 00  |...self...1.(...|

In this case, you use the command-line tool hexdump to view the contents of your binary file, but it’s rather difficult to understand. While there are bits of text here and there, much of the data looks like gibberish.

Another great advantage of a textual data format is that text has a uniform and mostly unambiguous interpretation. As long as you know the correct character encoding, which is almost always the ubiquitous UTF-8 these days, then you’ll be able to read your serialized messages everywhere on any hardware and system imaginable. No wonder so many popular serialization formats like XML, JSON, YAML, and CSV are all based on text.

Unfortunately, there’s no rose without a thorn. Text-based data serialization formats are slower to process than their binary counterparts because text has to be translated to native data types. They also use the available space less efficiently, which can limit their usefulness in larger datasets. Therefore, textual formats aren’t as common in data science and other fields that have to deal with big data.

Moreover, textual formats tend to become overly verbose and may contain a lot of redundant boilerplate. Notice how each element in the Atom feed above, such as the author of the feed, is wrapped in at least one pair of opening and closing XML tags, adding to the bloat. You can try compressing the serialized data to mitigate that, but it’ll add some overhead without providing predictable message sizes.

If you’re able to describe your data using plain English, then a textual data format is all you need. But what if you need to mix both text and binary assets in one text message? It’s possible. After all, email uses a text-based protocol, yet it lets you include binary attachments like pictures or PDF documents.

Embedding binary data in a text-based format requires expressing arbitrary bytes using a limited number of characters. One such technique involves Base64 encoding, which turns bytes into ASCII characters. The downside is that it increases the size of the binary asset by about thirty percent. So, sharing your vacation photos with colleagues at work through email can quickly bring your department’s network down!

There’s also the issue of security risks associated with leaking sensitive information due to the ease of reading textual data formats. In contrast, making sense of a message serialized to a binary data format is more challenging but certainly not impossible. Therefore, serializing private information like passwords or credit card numbers always requires strong encryption regardless of the data format!

To sum up, here’s how textual and binary data formats stack up against each other:

With this table, you can understand the key differences between textual and binary data serialization formats before deciding which one fits your specific needs. Once you know the answer, the next question should be whether to use a schema or not.

Schemaless vs Schema-Based

Regardless of whether they’re textual or binary, many data serialization formats require a schema document, which is a formal description of the expected structure of serialized data. At the same time, some formats are schemaless, while others can work with or without a schema:

Depending on the format at hand, you can express the corresponding schema differently. For example, it’s common to provide an XML-based XSD schema for XML documents, while the binary Avro format relies on JSON for its schemas. Protocol Buffers use their own interface definition language (IDL), on the other hand.

The schema for the Atom feed that you saw earlier leverages a somewhat dated RELAX NG format, which stands for regular language for XML next generation. Unlike the more widespread XML Schema (XSD), it’s not based on XML itself:

# -*- rnc -*-
# RELAX NG Compact Syntax Grammar for the
# Atom Format Specification Version 11

namespace atom = "http://www.w3.org/2005/Atom"
namespace xhtml = "http://www.w3.org/1999/xhtml"
namespace s = "http://www.ascc.net/xml/schematron"
namespace local = ""

start = atomFeed | atomEntry

# Common attributes

atomCommonAttributes =
   attribute xml:base { atomUri }?,
   attribute xml:lang { atomLanguageTag }?,
   undefinedAttribute*

# ...

A schema typically defines the allowed set of elements and attributes, their arrangement, relationships, and the associated constraints, such as whether an element is required or what range of values it can take. You can think of it as the vocabulary and grammar of a data serialization language.

This concept is analogous to a relational database schema, which specifies the tables, their column types, foreign keys, and so on. It’s a blueprint for re-creating the database from scratch, which can also act as a form of documentation. At runtime, the database schema governs your data’s referential integrity. Lastly, it helps facilitate object-relational mapping (ORM) in frameworks like Django.

The benefits of using a schema in a data serialization format include:

• Automation: The formal specification of data allows you to generate code stubs in different programming languages to handle automatic serialization and deserialization of each language’s native data types. This is also known as data binding when you use the XML format.
• Consistency: The schema enforces a standard structure for serialized data, ensuring its integrity and consistency across different systems.
• Documentation: The schema provides a clear definition of the structure of the data, which can help you quickly understand how the information is organized.
• Efficiency: Referencing the schema instead of including explicit field names reduces the size of serialized data. The schema can be known ahead of time or be embedded in the serialized message.
• Interoperability: Sharing the schema between different applications or services can facilitate their integration by allowing them to communicate with each other.
• Validation: The schema can be used to validate serialized data in an automated way, catching potential errors early on.

While there are many benefits to employing a schema, it also comes with some drawbacks, which you should weigh before deciding whether or not to use one. The highest price to pay for adopting a schema is the limited flexibility of the serialized data. The schema enforces a rigid structure, which may not be desirable if your data evolves over time or is dynamic in the sense that it doesn’t have a fixed layout.

Moreover, it can be difficult to alter a schema once you commit to one. Even though some schema-based formats, such as Avro, allow for schema versioning, this may generally break the consistency of data that you previously serialized. To mitigate this problem, some tools in the relational database domain offer schema migration, which gradually transforms data from an old schema to a new one.

For rapid prototyping or when working with unstructured data with unpredictable layouts, a schemaless data serialization format may be more suitable. Conceptually, this is like having a NoSQL database that can accept and process data from multiple sources. New types of elements or unknown attributes would be ignored without breaking the system instead of failing the schema validation.

All in all, these are the most important pros and cons of schemaless and schema-based data serialization formats:

All right, you’ve successfully narrowed down your options regarding the numerous types of data serialization formats. You’ve determined whether choosing a binary format over a textual one is more suitable in your case. Additionally, you understand when to use a schema-based format. Nonetheless, there are still a few choices left on the table, so you must ask yourself some more questions.

General-Purpose vs Specialized

Some data serialization formats are expressive enough to represent arbitrary data, making them universal formats. For example, JSON has become the prevailing data serialization format, especially in web development and REST API design. This format has a minimalistic syntax with only a few essential building blocks that are straightforward to map onto numerous data types in the vast landscape of programming languages.

On the opposite end of the spectrum, you’ll find specialized formats that can only represent a particular type of data. For example, XML continues to be an excellent format for describing deeply nested hierarchical data like user interfaces. After all, XML belongs to the same family of markup languages as HTML, which is widely used to structure content on the Internet.

Another example of a specialized data serialization format is the comma-separated values (CSV) format. It works best with flat tabular data like spreadsheets, database tables, or DataFrames. While you might be able to serialize a data record by mapping its attributes to table columns, modeling hierarchical data with CSV is less convenient than with XML. But, as a textual format, CSV can reach its limits even when handling tabular data.

In data science, you often have to process enormous amounts of data. To optimize performance and reduce storage costs, it’s usually preferable to choose a binary data serialization format dedicated to such large datasets.

These days, Parquet and Feather are gaining popularity in the data science space. They’re both compatible with Arrow, which is an in-memory specification that allows data-sharing between different libraries and even different programming languages. A couple of older but still popular ones are HDF5 and NetCDF. Their newer counterpart, Zarr, offers better support for distributed data storage and computation.

Special data serialization formats emerged in other domains, as well. Some examples include the following:

• DICOM: A binary format for storing and transmitting medical images
• GeoJSON: A specialized flavor of JSON for serializing geographic features
• GPX: An XML-based format for exchanging GPS coordinates
• MusicXML: An XML-based format for storing musical notation
• OBJ: A textual format for storing three-dimensional models

Whether you can use a textual or binary format, with or without a schema, may actually depend on your use case. Some specialized data serialization formats give you little choice in that regard. But there’s one final question that you must ask yourself before choosing the right data serialization format for you. You’ll read about it in the next section.

Portable vs Python-Specific

Another criterion to consider when choosing a data serialization format for your use case is where you’re going to use it. If you wish to exchange information between foreign systems, then opt for a popular data serialization format that’s globally understood. For example, JSON and Protocol Buffers are widely adopted across different programming languages and platforms.

On the other hand, if you only intend to serialize and deserialize data within Python, then you may choose a Python-specific format for practicality reasons. It’ll be a more efficient and convenient option, provided that you’re not planning to share the serialized data with other systems.

Python ships with the following modules in the standard library, which provide binary data serialization formats for different purposes:

• pickle: Python object serialization
• marshal: Internal object serialization
• shelve: Python object persistence
• dbm: An interface to Unix databases

In practice, you’ll almost always want to serialize your objects with pickle, which is the standard data serialization format in Python. The rest on the list are either low-level formats used internally by the interpreter or legacy formats kept for compatibility. You’ll review them now to get a complete picture of the available options.

Python uses marshal behind the scenes to read and write special files containing the bytecode of the imported modules. When you import a module for the first time, the interpreter builds a corresponding .pyc file with compiled instructions to speed up subsequent imports. Here’s a short code snippet that roughly demonstrates what happens under the hood when you import a module:

>>> import marshal

>>> def import_pyc(path):
...     with path.open(mode="rb") as file:
...         _ = file.read(16)  # Skip the file header
...         code = marshal.loads(file.read())
...         exec(code, globals())
...
>>> import sysconfig
>>> from pathlib import Path

>>> cache_dir = Path(sysconfig.get_path("stdlib")) / "__pycache__"
>>> module_path = cache_dir / "decimal.cpython-312.pyc"

>>> import_pyc(module_path)
>>> Decimal(3.14)
Decimal('3.140000000000000124344978758017532527446746826171875')

First, you locate the compiled .pyc file of the decimal module in Python’s standard library and then use marshal to execute its bytecode. As a result, you can access the Decimal class in the highlighted line without having to import it explicitly.

The marshal module is fairly limited in functionality. It can only serialize a few primitive values like integers and strings, but not user-defined or built-in classes. The implementation details of the format are left intentionally undocumented to deter you from using it. The module authors don’t give any guarantees about its backward compatibility. Your marshaled data could become unreadable or incompatible with newer Python versions one day.

The official documentation makes it clear that you’re not supposed to use marshal in your code. Again, you should generally prefer pickle to transmit or persistently store Python objects. But sometimes, you want a simple key-value store for your objects, which might be the case for caching purposes. In such a case, you can take advantage of shelve, which combines pickle and a lower-level dbm module:

>>> import shelve
>>> with shelve.open("/tmp/cache.db") as shelf:
...     shelf["last_updated"] = 1696846049.8469703
...     shelf["user_sessions"] = {
...         "jdoe@domain.com": {
...             "user_id": 4185395169,
...             "roles": {"admin", "editor"},
...             "preferences": {
...                 "language": "en_US",
...                 "dark_theme": False
...             }
...         }
...     }
...

The resulting shelf is a dictionary-like object or, more specifically, a hash table whose keys must be Python strings and whose values must be picklable objects. The module automatically calls .encode() on the keys to apply the default character encoding and uses pickle to convert the corresponding values to byte sequences.

Because shelve is a convenience module that builds on top of pickle and dbm, you can use the latter to take a peek at what’s inside the created file:

>>> import dbm
>>> with dbm.open("/tmp/cache.db") as db:
...     for key in db.keys():
...         print(f"{key} = {db[key]}")
...
b'user_sessions' = b'\x80\x04\x95{\x00\x00\x00\x00\x00\x00\x00}…'
b'last_updated' = b'\x80\x04\x95\n\x00\x00\x00\x00\x00\x00\x00G…'

Both keys and values in the file are stored as raw bytes. Additionally, the values comply with the pickle protocol, which shelve uses internally to serialize Python objects.

Python’s dbm module is an interface to DBM databases popular on Unix. For instance, the manual pages on Unix-like operating systems use this format to index the individual documentation pages. Note that a DBM instance only supports a few basic operations of the regular Python dictionary, so it might not expose methods like .copy() or support the union operators and iteration.

If you have special needs that the standard pickle format can’t fulfill, then consider installing a third-party library like dill, which extends its capabilities while remaining compatible with pickle. It can handle more sophisticated data types, including lambda expressions and Python modules.

One key point to keep in mind when choosing a Python-specific serialization format, regardless of which one, is the security ramifications. Because serialized data may contain malicious code that could compromise or damage your system, you should only deserialize data from trusted sources! This applies to pickle and other formats that allow arbitrary code execution during deserialization.

As you can see, there’s no one-size-fits-all data serialization format, as each comes with its own pros and cons. While this section gave you a comprehensive overview of the various categories, it’s now time dive deeper into concrete formats to understand them better.

Pickle Your Python Objects

As you’ve learned, pickle is the standard data serialization format in Python, and it’s capable of representing a wide range of native data types as well as user-defined classes. It’s a binary format specific to Python, which requires no schema definition and can handle data in almost any shape or form. If you need to persist your Python objects for later use or send them to another Python interpreter, then this should be your preferred choice.

Getting started with pickle is fairly straightforward:

>>> import pickle

>>> data = 255
>>> with open("filename.pkl", mode="wb") as file:
...     pickle.dump(data, file)
...

>>> pickle.dumps(data)
b'\x80\x04K\xff.'

First, you need to import the pickle module and then call either dump() or dumps() to turn an arbitrary Python object into a bytes instance. The first function expects a file-like object open in binary mode for writing, and the latter returns the sequence of bytes to the caller. Both functions allow you to choose the underlying protocol version, which might produce slightly different output:

>>> for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
...     print(f"v{protocol}:", pickle.dumps(data, protocol))
...
v0: b'I255\n.'
v1: b'K\xff.'
v2: b'\x80\x02K\xff.'
v3: b'\x80\x03K\xff.'
v4: b'\x80\x04K\xff.'
v5: b'\x80\x05K\xff.'

>>> pickle.DEFAULT_PROTOCOL
4

At the time of writing, protocol version 5 was the highest, and version 4 was the default. In general, higher versions are more efficient but may not work in older Python releases. Therefore, it’s usually good practice to stick with the relatively conservative defaults if you want your code to remain portable across Python versions. You can always try compressing the resulting byte stream to reduce its size.

Deserializing pickled data boils down to calling the module’s load() or loads(), which are counterparts of the two functions mentioned earlier:

>>> with open("filename.pkl", mode="rb") as file:
...     pickle.load(file)
...
255

>>> pickle.loads(b"\x80\x04K\xff.")
255

When unpickling data, you don’t get the option to specify the protocol version because it gets baked into the pickled byte sequence during serialization. This ensures that you can correctly unpickle your data using the right protocol.

>>> import array
>>> import pickle
>>> pickle.loads(array.array("B", [128, 4, 75, 255, 46]))
255

The pickle module can deal with most data types that you’ll ever work with. Moreover, it can handle objects with reference cycles, such as graphs, including recursive data structures and even deeply nested ones up to the recursion limit in Python:

>>> cycle = {}
>>> cycle["sibling"] = {"sibling": cycle}
>>> pickle.loads(pickle.dumps(cycle))
{'sibling': {'sibling': {...}}}

>>> recursive = []
>>> recursive.append(recursive)
>>> pickle.loads(pickle.dumps(recursive))
[[...]]

>>> import sys
>>> deeply_nested = []
>>> for _ in range(sys.getrecursionlimit() // 2):
...     deeply_nested = [deeply_nested]
...
>>> pickle.loads(pickle.dumps(deeply_nested))
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[...]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]

The variable cycle in the code snippet above is a dictionary whose only key-value pair holds another dictionary, which in turn contains a reference back to the original dictionary, forming a cycle. The recursive list above has only one element, which is a reference to itself. Finally, the deeply_nested list uses half of Python’s maximum recursion limit to wrap lists inside one another like a Matryoshka doll.

However, there are a few notable exceptions that will cause an error when you try to pickle an object. In particular, instances of the following data types aren’t picklable by design:

• Very deeply nested data structures approaching Python’s recursion limit
• Lambda expressions
• Generator objects
• Python modules
• File objects
• Network sockets
• Database connections
• Threads
• Stack frames

For example, these are the kinds of errors that you’ll get when you try to pickle a lambda expression, a Python module, and a very deeply nested list:

>>> pickle.dumps(lambda: None)
Traceback (most recent call last):
  ...
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7fdd14107920>:
⮑ attribute lookup <lambda> on __main__ failed

>>> pickle.dumps(pickle)
Traceback (most recent call last):
  ...
TypeError: cannot pickle 'module' object

>>> very_deeply_nested = []
>>> for _ in range(sys.getrecursionlimit()):
...     very_deeply_nested = [very_deeply_nested]
...
>>> pickle.dumps(very_deeply_nested)
Traceback (most recent call last):
  ...
RecursionError: maximum recursion depth exceeded while pickling an object

This limitation particularly affects inter-process communication (IPC) in Python, which uses pickle to serialize and deserialize data exchanged between the running interpreter processes:

from concurrent.futures import ProcessPoolExecutor

def worker(function, parameter):
    return function(parameter)

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        future = pool.submit(worker, lambda x: x**2, 10)
        future.result()

This code will result in a pickling error because of the attempt to send a lambda expression to another system process. When you use a process pool from concurrent.futures or the equivalent pool from the multiprocessing module, then you won’t be able to share certain types of data between your workers running in parallel.

Similarly, copying objects in Python relies on the pickle module, which suffers from the same deficiencies:

>>> def generator():
...     yield
...

>>> import copy
>>> copy.copy(generator())
Traceback (most recent call last):
  ...
TypeError: cannot pickle 'generator' object

If that’s too prohibitive for your needs, then consider using the third-party dill package, which acts as a drop-in replacement for pickle. It has the same interface as pickle and adheres to its underlying protocol while extending it to some extent:

>>> import pickle
>>> pickle.dumps(42)
b'\x80\x04K*.'

>>> import dill
>>> dill.dumps(42)
b'\x80\x04K*.'

>>> dill.loads(dill.dumps(lambda: None))
<function <lambda> at 0x7f78e793e660>

>>> dill.dumps(generator())
Traceback (most recent call last):
  ...
TypeError: cannot pickle 'generator' object

For basic data types, dill behaves similarly to pickle but lets you serialize some of the more exotic types, such as lambda expressions. At the same time, it can’t serialize everything—including generator objects.

try:
    import dill as pickle
except ModuleNotFoundError:
    import pickle

Anyway, as long as you stay within most of the native data types in Python, classes from the standard library, or even classes of your own, then pickle is all you need. However, in some cases, you’ll want more control over the serialization and deserialization processes.

Customize the Pickling Process

When you serialize a custom object, Python uses type introspection to discover the underlying class attributes and instance attributes. Whether these attributes are stored in .__dict__ or .__slots__, the pickle module can correctly recognize them. Their names and the corresponding values will be encoded on the target byte stream along with the fully qualified name of your class, which includes the module name where you defined the class.

Deserializing a pickled class instance resembles cloning an object, as it bypasses the initializer method completely. This may not be ideal when your initializer method contains important setup code for the object.

Other situations that may require customizing the pickling or unpickling process include:

• Adding metadata: During pickling, you can introduce extra attributes that aren’t part of your object to augment the serialized byte stream with additional information, like timestamps.
• Hiding sensitive information: To avoid serializing passwords and other secrets, you can exclude one or more attributes from pickling.
• Unpickling stateful objects: If your object has a non-picklable state, such as a database connection or an open file that can’t be directly serialized, then you might need to include some extra logic to handle the state during pickling and unpickling.
• Providing default values: When the structure of your class evolves, it may gain new attributes that old instances lack. In such cases, you can provide default values for missing attributes when unpickling old objects.
• Reducing pickled size: During unpickling, you can recalculate some attributes instead of persisting them on the byte stream if they take a lot of space in serialized form.

To hook into the pickling process, you can specify these two special methods in your class:

1. .__getstate__()
2. .__setstate__()

Python will call the first one before pickling an instance of your class, expecting the method to return a dictionary of attribute names and their values. Conversely, during unpickling, Python will call the second method and pass a dictionary of attributes to let you properly initialize your object after its state has been deserialized.

Say that you have the following data class representing a user in your system, which has the .name and .password attributes:

 1import time
 2from dataclasses import dataclass
 3
 4@dataclass
 5class User:
 6    name: str
 7    password: str
 8
 9    def __getstate__(self):
10        state = self.__dict__.copy()
11        state["timestamp"] = int(time.time())
12        del state["password"]
13        return state
14
15    def __setstate__(self, state):
16        self.__dict__.update(state)
17        with open("/dev/random", mode="rb") as file:
18            self.password = file.read(8).decode("ascii", errors="ignore")

To control the pickling and unpickling of your instances, you defined the two special methods above. Here’s how they work line by line:

• Line 10 makes a copy of the object’s internal dictionary of attributes comprising the user’s name and password.
• Line 11 injects an extra attribute named timestamp with the current Unix timestamp into the copied dictionary.
• Line 12 removes the key-value pair corresponding to the user’s password from the new dictionary.
• Line 13 returns the new dictionary with the modified state of your object to the pickle module.
• Line 16 populates the object’s internal dictionary using the deserialized state provided by the pickle module.
• Lines 17 and 18 set a new password based on random bytes loaded from the /dev/random file on a Unix-like operating system.

As you can see, pickle leaves the door open to overriding its default behavior, giving you plenty of flexibility. If you need even more freedom, then consider defining a custom Pickler object to serialize references of objects persisted externally—for example, in a database. You can also create a private dispatch table to decide when to use a custom serialization code and when to fall back to the default one.

With the protocol version 5 proposed and described in PEP 574, you can now efficiently pickle out-of-bound data. This can help serialize very large datasets, such as NumPy arrays or pandas DataFrames, meant for transferring between processes or distributed machines rather than persisting on disk. By leveraging buffers managed outside the usual pickling process, data engineers can avoid unnecessary copies and significantly reduce memory use.

So far, you’ve seen examples of serializing Python objects to a byte stream using the standard pickle module. It’s the most versatile, efficient, and battle-proven method for object serialization in Python. But it won’t always be the best choice due to its lack of cross-language compatibility and potential security risks, which you’ll learn how to mitigate later.

And now, it’s time to look into serializing Python objects using the text-based JSON format, which enjoys widespread use beyond the Python ecosystem.

Encode Objects Using JSON

Unlike the binary protocols that the pickle module uses, JSON is a textual serialization format readable by humans. Thanks to its huge popularity and simplicity, it’s become a universally accepted standard for data exchange across various programming languages and platforms.

Unfortunately, you can only represent a few Python data types with pure JSON, whereas pickle is able to handle nearly all Python data types out of the box, including more advanced cases like user-defined classes. That said, you’ll learn how to customize the serialization of various types with JSON in this section.

The interface of the json module looks similar to pickle and other serialization-related modules like PyYAML. You get the familiar dump() and dumps() methods, along with their load() and loads() counterparts.

In the following example, you dump a Python object to a JSON string:

>>> import json

>>> data = {
...     "email": None,
...     "name": "John Doe",
...     "age": 42.5,
...     "married": True,
...     "children": ["Alice", "Bob"],
... }

>>> print(json.dumps(data, indent=4, sort_keys=True))
{
    "age": 42.5,
    "children": [
        "Alice",
        "Bob"
    ],
    "email": null,
    "married": true,
    "name": "John Doe"
}

In this case, you serialize a Python dictionary whose keys are strings mapped to a few different data types. What you get as a result is a Python string formatted according to the grammar rules of JSON. Notice that you can optionally request to pretty-print the output and sort the keys alphabetically to improve the readability of larger objects.

The JSON format supports only six native data types:

1. Array: [1, 2, 3]
2. Boolean: true, false
3. Null: null
4. Number: 42, 3.14
5. Object: {"key1": "value", "key2": 42}
6. String: "Hello, World!"

In some cases, the json module will be able to convert your Python values to their closest JSON equivalents. At other times, it’ll fail with the following error:

>>> json.dumps({"Saturday", "Sunday"})
Traceback (most recent call last):
  ...
TypeError: Object of type set is not JSON serializable

The error message above tells you that a Python set isn’t serializable using the JSON format. However, you can teach the json module how to deal with such non-standard data types or even custom classes.

There are two ways to do so. You can provide callback functions to the dumping and loading functions, or you can extend the encoder and decoder classes from the json module.

To serialize a Python set as a JSON string using the first approach, you could define the following callback function and pass its reference to json.dump() or json.dumps() through the default parameter:

>>> def serialize_custom(value):
...     if isinstance(value, set):
...         return {
...             "type": "set",
...             "elements": list(value)
...         }
...

>>> data = {"weekend_days": {"Saturday", "Sunday"}}
>>> json.dumps(data, default=serialize_custom)
'{"weekend_days": {"type": "set", "elements": ["Sunday", "Saturday"]}}'

Python will now call serialize_custom() for every object that it can’t serialize as JSON by itself. It’ll pass down that object as an argument to your function, which should return one of the known data types. In this case, the function represents a Python set as a dictionary with elements encoded as a list and a type field denoting the intended Python data type.

If you’re happy with this new representation, then you’re essentially done. Otherwise, you can implement the corresponding deserialization callback to retrieve the original Python set from its current JSON representation:

>>> def deserialize_custom(value):
...     match value:
...         case {"type": "set", "elements": elements}:
...             return set(elements)
...         case _:
...             return value
...
>>> json_string = """
...     {
...         "weekend_days": {
...             "type": "set",
...             "elements": ["Sunday", "Saturday"]
...         }
...     }
... """

>>> json.loads(json_string, object_hook=deserialize_custom)
{'weekend_days': {'Sunday', 'Saturday'}}

Here, you pass a reference to deserialize_custom() through the object_hook parameter of json.loads(). Your custom function takes a Python dictionary as an argument, letting you convert it into an object of your choice. Using structural pattern matching, you identify if the dictionary describes a set, and if so, return a Python set instance. Otherwise, you return the dictionary as is.

In addition to these generic callbacks, you can specify a few specialized ones to override how to deserialize integer and floating-point number literals in JSON:

>>> json.loads("42", parse_int=float)
42.0

>>> from decimal import Decimal
>>> json.loads("3.14", parse_float=Decimal)
Decimal('3.14')

>>> json.loads("[NaN, Infinity, -Infinity]", parse_constant=str)
['NaN', 'Infinity', '-Infinity']

In the code snippet above, you convert all integers to Python floats and all floating-point number literals to decimal numbers. You also treat the special constants defined in the IEEE 754 standard, such as NaN, as regular strings.

If you prefer a more object-oriented approach to customizing the logic behind JSON serialization in Python, then extend the JSONEncoder and JSONDecoder classes. They’ll let you keep your reusable code in one place while giving you even more fine-grained control, including a streaming API. However, the general principles are the same as with the callback-based approach.

When you decide to use JSON over the binary pickle format, remember that you’ll have to take care of many corner cases manually. One such area is the handling of reference cycles, which pickle accounts for automatically:

>>> cycle = {}
>>> cycle["sibling"] = {"sibling": cycle}

>>> json.dumps(cycle)
Traceback (most recent call last):
  ...
ValueError: Circular reference detected

>>> json.dumps(cycle, check_circular=False)
Traceback (most recent call last):
  ...
RecursionError: maximum recursion depth exceeded while encoding a JSON object

When an object contains a cycle, the pure json module is unable to serialize it. You must implement some mechanism of tracking the objects that you’ve already serialized and reconstruct the original object graph in memory during deserialization.

This exhausts the list of data serialization formats supported natively by Python. Next, you’ll look into some third-party alternatives.

Use Formats Foreign to Python

Both pickle and JSON have their pros and cons. The first one can automatically serialize many types of Python objects out of the box but isn’t portable or human-readable. On the other hand, the text-based JSON is widely supported outside of Python but comes with only a few basic data types. Fortunately, you can combine the best of both worlds with the help of the jsonpickle library.

Contrary to what the name suggests, jsonpickle doesn’t actually use pickle under the hood to serialize Python objects. That would defeat the purpose of creating a more portable and human-readable format. Instead, it uses a custom serialization format on top of JSON to allow you to serialize complex Python objects.

If you still have your User data class around, then this is how you can serialize it to a JSON-formatted string with jsonpickle:

>>> import jsonpickle
>>> from models import User
>>> user = User(name="John", password="*%!U8n9erx@GdqK(@J")
>>> user_json = jsonpickle.dumps(user, indent=4)
>>> print(user_json)
{
    "py/object": "models.User",
    "py/state": {
        "name": "John",
        "timestamp": 1699805990
    }
}

>>> jsonpickle.loads(user_json)
User(name='John', password='\\\x06,\x19')

As you can see, the library respects the standard .__getstate__() and .__setstate__() methods that pickle uses. Moreover, it retains the type’s fully qualified name as a plain string in the resulting JSON under the "py/object" key. This means that the corresponding class must be importable when you try deserializing one of its instances.

That idea sounds remarkably similar to application-specific tags that some YAML parsers use to extend their capabilities. For example, this is what the same user object will look like when you serialize it using the PyYAML library:

>>> print(yaml.dump(user))
!!python/object:models.User
name: John
timestamp: 1699807790

The first line, which starts with a double exclamation mark (!!), uses a custom tag to let you store a User object. YAML is an excellent data serialization format because it supports a large number of data types that you’ll find in many programming languages. Even so, you can extend it with custom data types like the User class above for your specific needs.

Now that you know how to serialize Python objects representing data, you can tackle the task of serializing executable Python code.

Pickle Importable Symbols

In Python, modules, functions, and classes are the most fundamental building blocks of any program, encapsulating its logic. The pickle module can serialize functions and classes but with certain limitations.

If you followed along with the earlier section on customizing the pickling process, then you might recall that pickle only retains the names of your functions and classes. Later, it uses those names to look up the corresponding source code in your virtual environment.

Say you have a Python module with the following definitions, which you’d like to serialize using pickle:

def create_plus(x):
    def plus(y):
        return x + y

    return plus

plus_one = create_plus(1)
plus_two = lambda x: x + 2

The create_plus() function is an example of a higher-order function, or more specifically, a factory function that returns a brand-new function closure created dynamically at runtime. The outer function takes a number, x, as input and returns an inner function, which takes another number, y, as an argument and adds that number to the first one.

The two variables at the bottom, plus_one and plus_two, represent callable objects. The first is a function closure returned by the factory function defined earlier, while the other one is a lambda expression.

Now, because Python modules aren’t picklable, you can’t serialize the entire module using pickle:

>>> import pickle
>>> import plus
>>> pickle.dumps(plus)
Traceback (most recent call last):
  ...
TypeError: cannot pickle 'module' object

By the same token, you can’t serialize inner functions and lambda expressions imported from your module, as well as code objects of regular functions. To get the code object associated with your function, you can access its .__code__ attribute, as shown at the bottom of the following code snippet:

>>> pickle.dumps(plus.plus_one)
Traceback (most recent call last):
  ...
AttributeError: Can't pickle local object 'create_plus.<locals>.plus'

>>> pickle.dumps(plus.plus_two)
Traceback (most recent call last):
  ...
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f6f32f2a480>:
⮑ attribute lookup <lambda> on plus failed

>>> pickle.dumps(plus.create_plus.__code__)
Traceback (most recent call last):
  ...
TypeError: cannot pickle code objects

The first error message tells you that you can’t pickle a local object, which is a function defined inside another function in this case. Python must be able to reference your object from the global namespace to serialize it. Similarly, a lambda is an anonymous function defined on the fly as an expression that doesn’t belong to the global scope.

You can only pickle top-level functions, such as create_plus(), that you defined with the def keyword in the global namespace of a module:

>>> pickle.dumps(plus.create_plus)
b'\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c
⮑ \x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94.'

Notice that the resulting byte stream is fairly short and contains the fully qualified name of your function, including the enclosing module name, plus. You can now store this byte sequence on disk or transfer it to another Python interpreter over a network. However, if the receiving end has no access to the corresponding functions or class definitions, then you still won’t be able to unpickle the serialized code:

>>> # This runs on a remote computer without the "plus" module
>>> import pickle
>>> pickle.loads(
...     b"\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c"
...     b"\x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94."
... )
Traceback (most recent call last):
  ...
ModuleNotFoundError: No module named 'plus'

As you can see, Python is able to deserialize your byte stream correctly, knowing which module to import, but it fails to find the missing plus module.

In contrast, when the enclosing module of your serialized function or class is on the import path, making the module importable, then you’ll be able to retrieve the original object from the byte stream:

>>> # This runs where the "plus" module is importable
>>> import pickle
>>> pickle.loads(
...     b"\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c"
...     b"\x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94."
... )
<function create_plus at 0x7f89d84a2700>

Here, you get a reference to the create_plus() function, which Python imported for you behind the scenes. It’s as if you had imported it yourself.

The lack of a built-in mechanism for serializing Python source code is a bit disappointing. A quick and dirty way to work around this limitation of the pickle module is to share the code in literal form for the exec() function to execute later:

>>> import importlib
>>> import inspect

>>> def get_module_source(module_name):
...     module = importlib.import_module(module_name)
...     return inspect.getsource(module)
...
>>> source_code = get_module_source("plus")
>>> print(source_code)
def create_plus(x):
    def plus(y):
        return x + y

    return plus

plus_one = create_plus(1)
plus_two = lambda x: x + 2

>>> exec(source_code)
>>> plus_two(3)
5

You define get_module_source() for dynamically importing a Python module by name with the help of importlib. Additionally, you rely on inspect to return the Python source code of the imported module as a string. Then, you use your new helper function to obtain the source code of the plus module that you defined earlier and execute the associated code with its function definitions. Finally, you call one of those functions without importing it.

This trick avoids serializing the code altogether, but it comes with its own set of problems. What if you only wanted to share a specific function or class without revealing the rest of your module? Perhaps executing the module causes unwanted side effects. Maybe you don’t have access to the high-level Python source code because it’s a compiled C extension module. Do you trust that the code will remain intact and secure when you execute it again?

>>> import dill.source
>>> from plus import plus_two
>>> dill.source.getsource(plus_two)
'plus_two = lambda x: x + 2\n'

The next few sections will address these concerns and provide alternative solutions to serializing executable code in Python using the third-party dill library.

Serialize Python Modules

You’ve already heard of the third-party dill library a few times in this tutorial. One of its main advantages over the standard library’s pickle module is the ability to serialize Python source code without requiring an importable enclosing module. However, there’s a catch that you need to be aware of, which you’ll learn about soon.

First of all, you can call dump_module(), which is specific to dill, in order to retain and later restore the state of global variables defined in a module. Unless you specify otherwise, dill serializes the special __main__ module by default. This can be useful when you’re experimenting in the interactive Python REPL and would like to continue the session where you left off, potentially on another computer:

>>> from dataclasses import dataclass

>>> @dataclass
... class Person:
...     first_name: str
...     last_name: str
...
>>> jdoe = Person("John", "Doe")

>>> import dill
>>> dill.dump_module("session.pkl")

Here, you define a data class, create an instance of it, and assign the resulting object to a global variable. Then, you dump the current session to a binary file, which you can load in a new Python REPL session later:

>>> import dill
>>> dill.load_module("session.pkl")

>>> jdoe
Person(first_name='John', last_name='Doe')

>>> Person("Jack", "Ryan")
Person(first_name='Jack', last_name='Ryan')

After loading the serialized module, you continue working with the object that you created before and the class that you defined earlier as if you’d never left the original session.

>>> import dill
>>> dill.load_module("plus.pkl")
Traceback (most recent call last):
  ...
ModuleNotFoundError: No module named 'plus'

Okay. Dumping Python modules with dill allows you to share your functions and classes between Python interpreters, but what if you wanted to be more selective about what you serialize? What about a situation where your recipient doesn’t have access to your source code? In such cases, you can use dill to dump a specific code object associated with a function or class.

Serialize Code Objects

Somewhat surprisingly, when you try serializing a function imported from another module, such as your create_plus() factory function from the plus module, dill and pickle may sometimes produce an identical binary sequence:

>>> import pickle
>>> import dill
>>> import plus

>>> pickle.dumps(plus.create_plus)
b'\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c
⮑ \x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94.'

>>> dill.dumps(plus.create_plus)
b'\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c
⮑ \x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94.'

This seems to break the promise made by dill to be capable of serializing such code without needing the enclosing module.

However, if you dump your function from within that module, then you’ll get the expected result. To do so, you can run your module through the Python interpreter with the -i option, which lets you inspect the finished program’s state in an interactive Python shell:

$ python -i plus.py
>>> import dill
>>> dill.dumps(create_plus)
b'\x80\x04\x95U\x01\x00\x00\x00\x00\x00\x00\x8c
⮑ \ndill._dill\x94\x8c\x10_create_function\x94
⮑ \x93\x94(h\x00\x8c\x0c_create_code\x94\x93
⮑ \x94(C\x04\x04\x01\n\x03\x94K\x01K\x00K\x00K
⮑ \x02K\x02K\x03C\x12\x87\x00\x97\x00\x88\x00f
⮑ \x01d\x01\x84\x08}\x01|\x01S\x00\x94Nh\x04(C
⮑ \x02\x04\x01\x94K\x01K\x00K\x00K\x01K\x02K
⮑ \x13C\x0e\x95\x01\x97\x00\x89\x01|\x00z\x00
⮑ \x00\x00S\x00\x94N\x85\x94)\x8c\x01y\x94\x85
⮑ \x94\x8c\x0c/tmp/plus.py\x94\x8c\x04plus\x94
⮑ \x8c\x19create_plus.<locals>.plus\x94K\x02C
⮑ \x0c\xf8\x80\x00\xd8\x0f\x10\x901\x89u\x88
⮑ \x0c\x94C\x00\x94\x8c\x01x\x94\x85\x94)t\x94R
⮑ \x94\x86\x94)h\x11h\r\x86\x94h\x0c\x8c
⮑ \x0bcreate_plus\x94h\x17K\x01C\x0f\xf8\x80\x00
⮑ \xf4\x02\x01\x05\x15\xf0\x06\x00\x0c\x10\x80K
⮑ \x94h\x10)h\x11\x85\x94t\x94R\x94c__builtin__
⮑ \n__main__\nh\x17NNt\x94R\x94}\x94}\x94\x8c
⮑ \x0f__annotations__\x94}\x94s\x86\x94b.'

With this option, Python executes the code in your module, including the function definitions, while letting you stay in the interactive shell to inspect global variables and other symbols afterward.

Because create_plus() is now a part of the global namespace, dill can serialize the function properly. It finally outputs a much longer byte sequence, which includes the serialized code object that you can deserialize without having the plus module in your virtual environment:

>>> # This runs on a remote computer without the "plus" module
>>> import dill
>>> create_plus = dill.loads(
...     b"\x80\x04\x95U\x01\x00\x00\x00\x00\x00\x00\x8c"
...     b"\ndill._dill\x94\x8c\x10_create_function\x94"
...     b"\x93\x94(h\x00\x8c\x0c_create_code\x94\x93"
...     b"\x94(C\x04\x04\x01\n\x03\x94K\x01K\x00K\x00K"
...     b"\x02K\x02K\x03C\x12\x87\x00\x97\x00\x88\x00f"
...     b"\x01d\x01\x84\x08}\x01|\x01S\x00\x94Nh\x04(C"
...     b"\x02\x04\x01\x94K\x01K\x00K\x00K\x01K\x02K"
...     b"\x13C\x0e\x95\x01\x97\x00\x89\x01|\x00z\x00"
...     b"\x00\x00S\x00\x94N\x85\x94)\x8c\x01y\x94\x85"
...     b"\x94\x8c\x0c/tmp/plus.py\x94\x8c\x04plus\x94"
...     b"\x8c\x19create_plus.<locals>.plus\x94K\x02C"
...     b"\x0c\xf8\x80\x00\xd8\x0f\x10\x901\x89u\x88"
...     b"\x0c\x94C\x00\x94\x8c\x01x\x94\x85\x94)t\x94R"
...     b"\x94\x86\x94)h\x11h\r\x86\x94h\x0c\x8c"
...     b"\x0bcreate_plus\x94h\x17K\x01C\x0f\xf8\x80\x00"
...     b"\xf4\x02\x01\x05\x15\xf0\x06\x00\x0c\x10\x80K"
...     b"\x94h\x10)h\x11\x85\x94t\x94R\x94c__builtin__"
...     b"\n__main__\nh\x17NNt\x94R\x94}\x94}\x94\x8c"
...     b"\x0f__annotations__\x94}\x94s\x86\x94b."
... )
>>> create_plus
<function create_plus at 0x7f0f88324220>
>>> plus_three = create_plus(3)
>>> plus_three
<function create_plus.<locals>.plus at 0x7f0f883242c0>
>>> plus_three(2)
5

When you pass the same byte sequence to dill.loads(), it returns a new function object that works exactly like your original create_plus() factory function. Even if the plus module with the corresponding Python source code can’t be found, the deserialized function still works as expected.

Inspecting a module to serialize one of its functions or classes is a lot of hassle. Sure, you could always slap the call to dill.dumps() at the bottom of your module and call it a day:

import dill

def create_plus(x):
    def plus(y):
        return x + y

    return plus

plus_one = create_plus(1)
plus_two = lambda x: x + 2

print(dill.dumps(create_plus))

This might work in some cases, but you generally prefer your code to be flexible so that you can decide later which function or class to serialize. Fortunately, you have another trick up your sleeve. You can manually substitute the special .__module__ attribute and instruct dill to recursively traverse the globals() dictionary:

>>> import dill
>>> import plus

>>> plus.create_plus.__module__
'plus'
>>> dill.dumps(plus.create_plus)
b'\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c
⮑ \x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94.'

>>> plus.create_plus.__module__ = None
>>> dill.dumps(plus.create_plus, recurse=True)
b'\x80\x04\x95N\x01\x00\x00\x00\x00\x00\x00\x8c
⮑ \ndill._dill\x94\x8c\x10_create_function\x94
⮑ \x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94
⮑ (C\x04\x04\x01\n\x03\x94K\x01K\x00K\x00K\x02K
⮑ \x02K\x03C\x12\x87\x00\x97\x00\x88\x00f\x01d
⮑ \x01\x84\x08}\x01|\x01S\x00\x94Nh\x04(C\x02
⮑ \x04\x01\x94K\x01K\x00K\x00K\x01K\x02K\x13C
⮑ \x0e\x95\x01\x97\x00\x89\x01|\x00z\x00\x00\x00S
⮑ \x00\x94N\x85\x94)\x8c\x01y\x94\x85\x94\x8c
⮑ \x0c/tmp/plus.py\x94\x8c\x04plus\x94\x8c\x19
⮑ create_plus.<locals>.plus\x94K\x02C\x0c\xf8\x80
⮑ \x00\xd8\x0f\x10\x901\x89u\x88\x0c\x94C\x00\x94
⮑ \x8c\x01x\x94\x85\x94)t\x94R\x94\x86\x94)h\x11h
⮑ \r\x86\x94h\x0c\x8c\x0bcreate_plus\x94h\x17K
⮑ \x01C\x0f\xf8\x80\x00\xf4\x02\x01\x05\x15\xf0
⮑ \x06\x00\x0c\x10\x80K\x94h\x10)h\x11\x85\x94t
⮑ \x94R\x94}\x94\x8c\x08__name__\x94Nsh\x17NNt
⮑ \x94R\x94}\x94}\x94\x8c\x0f__annotations__\x94}
⮑ \x94s\x86\x94b.'

The .__module__ attribute of a function or a class indicates the name of the Python module it belongs to. As you might expect, the initial value of the create_plus() function’s .__module__ attribute is plus. This triggers the fallback mechanism, which delegates the serialization to pickle, resulting in a short byte sequence comprising only the fully qualified name of your function instead of the code object.

By artificially changing this attribute’s value to None, you detach the function from its module. However, callable objects have another special attribute, .__globals__, which is a dictionary representing the global namespace where you defined them. Some of its elements may be redundant, forcing dill to serialize more than necessary and potentially making the enclosing module required for deserialization.

The .__globals__ attribute is read-only, so you can’t easily override it. What you can do, though, is instruct dill to recursively traverse that namespace and cherry-pick objects to serialize as needed. In contrast, when you remove the recurse=True parameter from one of the highlighted lines above, the resulting byte sequence will be a few times longer!

Finally, it’s worth noting that recursive functions need special attention when you deserialize them. Here’s a sample function, which calculates the nth Fibonacci number, represented as a byte stream produced by dill:

>>> import dill
>>> fn = dill.loads(
...     b"\x80\x04\x95-\x01\x00\x00\x00\x00\x00\x00\x8c\ndill._dill"
...     b"\x94\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c"
...     b"_create_code\x94\x93\x94(C\x02\x02\x01\x94K\x01K\x00K\x00K"
...     b"\x01K\x05K\x03CJ\x97\x00|\x00d\x01k\x02\x00\x00r\x02d\x02S"
...     b"\x00t\x01\x00\x00\x00\x00\x00\x00\x00\x00|\x00d\x01z\n\x00"
...     b"\x00\xab\x01\x00\x00\x00\x00\x00\x00t\x01\x00\x00\x00\x00"
...     b"\x00\x00\x00\x00|\x00d\x02z\n\x00\x00\xab\x01\x00\x00\x00"
...     b"\x00\x00\x00z\x00\x00\x00S\x00\x94NK\x02K\x01\x87\x94\x8c"
...     b"\x03fib\x94\x85\x94\x8c\x01n\x94\x85\x94\x8c\x0b/tmp/fib.py"
...     b"\x94h\x08h\x08K\x01C(\x80\x00\xd8\x10\x11\x90A\x92\x05\x881"
...     b"\xd0\x042\x9c3\x98q\xa01\x99u\x9b:\xac\x03\xa8A\xb0\x01\xa9E"
...     b"\xab\n\xd1\x1b2\xd0\x042\x94C\x00\x94))t\x94R\x94c__builtin__"
...     b"\n__main__\nh\x08NNt\x94R\x94}\x94}\x94\x8c\x0f__annotations__"
...     b"\x94}\x94s\x86\x94b."
... )
>>> fn
<function fib at 0x7fd6b68dc220>
>>> fn(5)
Traceback (most recent call last):
  ...
NameError: name 'fib' is not defined

You successfully load the function from that byte stream and assign it to a variable named fn. Later, you use your new variable to call the deserialized function, which fails with a NameError due to referencing an undefined symbol, fib. What’s going on?

A recursive function calls itself until the base condition is met. However, to call itself, the function must know its own name. In the serialized version, the function was named fib, as indicated by the code object that you evaluated in the Python REPL. But there’s no such name in your current scope.

To fix this problem, you can create an alias for the function, like so:

>>> fib = fn
>>> [fib(n) for n in range(10)]
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Great! With the function assigned to fib, the recursive calls now succeed because the correct function name exists in your scope.

You’ve covered a lot of ground when it comes to serializing executable code. Before closing this topic, you should address the elephant in the room, which is the potential security risk associated with deserializing code that can run on your computer.

Protect From Malicious Code

When you browse Python’s documentation of the various data serialization formats that allow you to serialize code, you’ll notice prominent warnings about the potential security risks involved. These can enable the execution of arbitrary code during deserialization, exposing you to hacker attacks.

The official documentation page of pickle demonstrates an example of a handmade byte stream, which results in running a system command when unpickled:

>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0

After unpickling this cleverly crafted byte sequence, Python automatically imports the os module and calls the system() function, passing a command to run in your shell. It merely executes the innocent echo command, which prints hello world on the screen. However, it could potentially do anything, including deleting files from your system, stealing sensitive data, installing malware, or even giving unauthorized access to your computer.

As a rule of thumb, you should always double-check that you can trust the source of the code before allowing it to run. But even that may not be enough when someone in the middle tampers with your code. The recommended practice is to either use a safer serialization format that requires input validation or implement countermeasures to prevent malicious code injection.

When using pickle, you can restrict which functions and classes are allowed to be deserialized by extending the Unpickler class and overriding its .find_class() method:

import importlib
import io
import pickle

class SafeUnpickler(pickle.Unpickler):
    ALLOWED = {
        "builtins": ["print"],
        "sysconfig": ["get_python_version"],
    }

    @classmethod
    def safe_loads(cls, serialized_data):
        file = io.BytesIO(serialized_data)
        return cls(file).load()

    def find_class(self, module_name, name):
        if module_name in self.ALLOWED:
            if name in self.ALLOWED[module_name]:
                module = importlib.import_module(module_name)
                return getattr(module, name)
        raise pickle.UnpicklingError(f"{module_name}.{name} is unsafe")

The SafeUnpickler class defined above contains a custom whitelist consisting of the only functions that you can deserialize. The .safe_loads() class method conveniently wraps the input byte sequence into a file-like object, which the unpickler expects. Finally, the overridden .find_class() method either calls one of the approved functions—returning its result to the caller—or raises an exception.

Any attempt to unpickle a function or class that’s not on your whitelist will fail. At the same time, you can execute deserialized code that’s been marked as safe:

>>> from safe_unpickler import SafeUnpickler

>>> SafeUnpickler.safe_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
  ...
_pickle.UnpicklingError: os.system is unsafe

>>> SafeUnpickler.safe_loads(b"cbuiltins\nprint\n(S'hello world'\ntR.")
hello world

>>> SafeUnpickler.safe_loads(b"csysconfig\nget_python_version\n(tR.")
'3.12'

By limiting what the pickle module can process, you provide a more secure way of deserializing executable code. But what if that’s too restrictive? Perhaps you really want to be able to call certain functions when the data is coming from a trusted source.

A common method of establishing trust is to digitally sign the serialized code with a shared secret key that only you and your recipient know. In Python, you can take advantage of the standard library module named hmac, which implements the HMAC algorithm used in cryptography for checking the authenticity and integrity of your messages.

Here’s how you can combine dill, hmac, and hashlib to make the serialization of executable code in Python trustworthy:

import hashlib
import hmac
import dill

def safe_dump(obj, file, secret_key):
    serialized_data = dill.dumps(obj)
    signature = sign(serialized_data, secret_key)
    dill.dump(signature, file)
    file.write(serialized_data)

def safe_load(file, secret_key):
    signature = dill.load(file)
    serialized_data = file.read()
    if signature == sign(serialized_data, secret_key):
        return dill.loads(serialized_data)
    raise dill.UnpicklingError("invalid digital signature")

def sign(message, secret_key, algorithm=hashlib.sha256):
    return hmac.new(secret_key, message, algorithm).digest()

The safe_dump() function takes an arbitrary Python object as the first argument, followed by a file-like object, which must be open in binary mode for writing, and a secret byte sequence used for signing. It then dumps the signature of your pickled data to the file using dill and writes the pickled data in the same file right after the signature.

Conversely, the safe_load() function expects a file-like object open for reading in binary mode and a secret byte sequence as arguments. It loads the signature from the file along with pickled data and calculates the signature for that data using the secret key supplied as an argument. If the two signatures compare equal, then the function unpickles and returns your data.

Both functions call the sign() helper to calculate the digital signature of your executable code. By default, the function uses the SHA-256 algorithm, which is popular for storing hashed passwords. However, you can replace it with any available cryptographic algorithm that fits your specific needs.

To verify if this works as expected, import and use your trustworthy module in a new Python shell:

>>> from pathlib import Path
>>> from trustworthy import safe_dump, safe_load

>>> path = Path("code.pkl")
>>> code = lambda a, b: a + b

>>> with path.open(mode="wb") as file:
...     safe_dump(code, file, b"top-secret")
...
>>> with path.open(mode="rb") as file:
...     safe_load(file, b"top-secret")
...
<function <lambda> at 0x7fa1248d0180>

You start by defining a path object to indicate the location of a binary file for your serialized code. The code that you’re going to serialize is a lambda expression stored in a variable named code. Then, you open the indicated file for writing and safely dump your code using top-secret as the key to sign the data with.

Later, you open the same file for reading and safely load its content using the correct secret key. As a result, you get a new callable object, which is a faithful replica of your original lambda expression.

Now, if someone tries to alter the code that you serialized, they’ll need to update the corresponding signature, which they can’t do without knowing the secret key. That’s why it’s important to keep the key secret! If you do, then you’ll immediately know when something’s wrong. The same is true when you supply an incorrect secret key:

>>> with path.open(mode="rb") as file:
...     safe_load(file, b"wrong-key")
...
Traceback (most recent call last):
  ...
_pickle.UnpicklingError: invalid digital signature

The key used to digitally sign the serialized data is different from the one used for deserialization. Therefore, the signatures don’t match, which should raise your suspicion.

Deserializing executable code often carries the risk of introducing potential security threats. If you don’t trust the source where the serialized data is coming from, then it’s best to use a safer format or introduce measures to prevent unauthorized code execution.

Build a REST API Using Flask

Flask is a popular web framework that gives you plenty of flexibility in terms of structuring your project and choosing its i