Welcome

Welcome to "Rust-Python interoperability"!
This course will teach you how to call Rust code from Python, packaged as a native extension module.

We assume you are familiar with both Rust and Python, but we don't assume any prior interoperability knowledge. We will provide a brief explanation and references whenever we rely on advanced language features in either language.

Methodology

This course is based on the "learn by doing" principle.
You'll build up your knowledge in small, manageable steps. It has been designed to be interactive and hands-on.

Mainmatter developed this course to be delivered in a classroom setting, over a whole day: each attendee advances through the lessons at their own pace, with an experienced instructor providing guidance, answering questions and diving deeper into the topics as needed.
If you're interested in attending one of our training sessions, or if you'd like to bring this course to your company, please get in touch.

You can also follow the course on your own, but we recommend you find a friend or a mentor to help you along the way should you get stuck. You can also find solutions to all exercises in the solutions branch of the GitHub repository.

Prerequisites

To follow this course, you must install:

  • Rust
  • rye, a Python package manager

If Rust is already installed on your machine, make sure to update it to the latest version:

# If you installed Rust using `rustup`, the recommended way,
# you can update to the latest stable toolchain with:
rustup update stable

These commands should successfully run on your machine:

cargo --version
rye --version

Don't start the course until you have these tools installed and working.

Structure

On the left side of the screen, you can see that the course is divided into sections.
To verify your understanding, each section is paired with an exercise that you need to solve.

You can find the exercises in the companion GitHub repository.
Before starting the course, make sure to clone the repository to your local machine:

# If you have an SSH key set up with GitHub
git clone git@github.com:mainmatter/rust-python-interoperability.git
# Otherwise, use the HTTPS URL:
#
#   git clone https://github.com/mainmatter/rust-python-interoperability.git

We recommend you work on a branch, so you can easily track your progress and pull updates from the main repository if needed:

cd rust-python-interoperability
git checkout -b my-solutions

All exercises are located in the exercises folder. Each exercise is structured as a Rust package. The package contains the exercise itself, instructions on what to do (in src/lib.rs), and a test suite to automatically verify your solution.

wr, the workshop runner

To verify your solutions, we've provided a tool that will guide you through the course. It is the wr CLI (short for "workshop runner"). Install it with:

cargo install --locked workshop-runner

In a new terminal, navigate back to the top-level folder of the repository. Run the wr command to start the course:

wr

wr will verify the solution to the current exercise.
Don't move on to the next section until you've solved the exercise for the current one.

We recommend committing your solutions to Git as you progress through the course, so you can easily track your progress and "restart" from a known point if needed.

Enjoy the course!

Author

This course was written by Luca Palmieri, Principal Engineering Consultant at Mainmatter.
Luca has been working with Rust since 2018, initially at TrueLayer and then at AWS.
Luca is the author of "Zero to Production in Rust", the go-to resource for learning how to build backend applications in Rust, and "100 Exercises to Learn Rust", a learn-by-doing introduction to Rust itself.
He is also the author and maintainer of a variety of open-source Rust projects, including cargo-chef, Pavex and wiremock.

Exercise

The exercise for this section is located in 01_intro/00_welcome

Anatomy of a Python extension

Don't jump ahead!
Complete the exercise for the previous section before you start this one.
It's located in exercises/01_intro/00_welcome, in the course GitHub's repository.
Use wr to start the course and verify your solutions.

To invoke Rust code from Python we need to create a Python extension module.

Rust, just like C and C++, compiles to native code. For this reason, extension modules written in Rust are often called native extensions. Throughout this course we'll use the terms Python extension, Python extension module and native extension interchangeably.

maturin

We'll use maturin to build, package and publish Python extensions written in Rust. Let's install it:

rye install "maturin>=1.6"

Exercise structure

All exercises in this course will follow the same structure:

  • an extension module written in Rust, in the root of the exercise directory
  • a Python package that invokes the functionality provided by the extension, in the sample subdirectory

The extension module will usually be tested from Python, in the sample/tests subdirectory. You will have to modify the Rust code in the extension module to make the tests pass.

Extension structure

Let's explore the structure of the extension module for this section.

01_setup
├── sample
├── src
│   └── lib.rs
├── Cargo.toml
└── pyproject.toml

Cargo.toml

The manifest file, Cargo.toml, looks like this:

[package]
name = "setup"
version = "0.1.0"
edition = "2021"

[lib]
name = "setup"
crate-type = ["cdylib"]

[dependencies]
pyo3 = "0.21.1"

Two things stand out in this file compared to a regular Rust project:

  • The crate-type attribute is set to cdylib.
  • The pyo3 crate is included as a dependency.

Let's cover these two points in more detail.

Linking

Static linking

By default, Rust libraries are compiled as static libraries.
All dependencies are linked into the final executable at compile-time, making the executable self-contained1.

That's great for distributing applications, but it's not ideal for Python extensions.
To perform static linking, the extension module would have to be compiled alongside the Python interpreter. Furthermore, you'd have to distribute the modified interpreter to all your users.
At the ecosystem level, this process would scale poorly, as each user needs to leverage several unrelated extensions at once. Every single project would have to compile its own bespoke Python interpreter.

Dynamic linking

To avoid this scenario, Python extensions are packaged as dynamic libraries.
The Python interpreter can load these libraries at runtime, without having to be recompiled. Instead of distributing a modified Python interpreter to all users, you must now distribute the extension module as a standalone file.

Rust supports dynamic linking, and it provides two different flavors of dynamic libraries: dylib and cdylib. dylib are Rust-flavored dynamic libraries, geared towards Rust-to-Rust dynamic linking. cdylib, on the other hand, are dynamic libraries that export a C-compatible interface (C dynamic library).

You need a common dialect to get two different languages to communicate with each other. They both need to speak it and understand it.
That bridge, today, is C's ABI (Application Binary Interface).

That's why, for Python extensions, you must use the cdylib crate type:

[lib]
crate-type = ["cdylib"]

pyo3

It's not enough to expose a C-compatible interface. You must also comply with the Python C API, the interface Python uses to interact with C extensions.

Doing this manually is error-prone and tedious. That's where the pyo3 crate comes in: it provides a safe and idiomatic way to write Python extensions in Rust, abstracting away the low-level details.

In lib.rs, you can see it in action:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyfunction]
fn it_works() -> bool {
    todo!()
}

/// A Python module implemented in Rust.
#[pymodule]
fn setup(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(it_works, m)?)?;
    Ok(())
}
}

We're using pyo3 to define a Python function, named it_works, that returns a boolean. The function is then exposed to Python at the top-level of our extension module, named setup.

That same function is then invoked from Python, inside sample/tests/test_sample.py:

from setup import it_works

def test_works(): 
    assert it_works()

We'll cover the details of #[pyfunction] and #[pymodule] in the next section, no worries.

pyproject.toml

Before we move on, let's take a look at pyproject.toml, the Python "manifest" of the extension module:

[build-system]
requires = ["maturin>=1.6,<2.0"]
build-backend = "maturin"

[project]
name = "setup"
# [...]
requires-python = ">=3.8"
dynamic = ["version"]
[tool.maturin]
features = ["pyo3/extension-module"]

It specifies the build system, the extension name and version, the required Python version, and the features to enable when building the extension module. This is what rye looks at when building the extension module, before delegating the build process to maturin, which in turn invokes cargo to compile the Rust code.

What do I need to do?

A lot has to go right behind the scenes to make a Python extension work.
That's why the exercise for this section is fairly boring—we want to verify that you can build and test a Python extension module without issues.

Things will get a lot more interesting over the coming sections, I promise!

Troubleshooting

You may run into this error when using rye and pyo3 together:

<compiler output>
= note: ld: warning: search path '/install/lib' not found
          ld: library 'python3.12' not found
          clang: error: linker command failed with exit code 1

This seems to be a bug in rye.
To work around the issue, run the following command in the root of the course repository:

cargo run -p "patcher"

wr should now be able to build the extension module without issues and run the tests. No linker errors should be surfaced.

The patcher tool is a temporary workaround for a bug in rye.
It hasn't been tested on Windows: please open an issue if you encounter any problems.

References

Footnotes

1

This is true up to an extent. In most cases, some dependencies are still dynamically linked, e.g. libc on most Unix systems. Nonetheless, the final executable is self-contained in the sense that it doesn't rely on the presence of the Rust standard library or any other Rust crate on the user's system.

Exercise

The exercise for this section is located in 01_intro/01_setup

Modules

In Python, just like in Rust, your code is organized into modules.
Your entire extension is a module!

That module is defined using pyo3's #[pymodule] procedural macro, as you've seen in the previous section:

#![allow(unused)]
fn main() {
#[pymodule]
fn setup(m: &Bound<'_, PyModule>) -> PyResult<()> {
   // [...]
}
}

setup becomes the entry point for the Python interpreter to load your extension.

Naming matters

The name of the annotated function is important: there must be at least one module with a name that matches the name of the dynamic library artifact that Python will try to load. This is the name of the library target specified in your Cargo.toml file:

[lib]
name = "name_of_your_rust_library"

If you don't have a [lib] section, it defaults to the name of your package, specified in the [package] section.

If the module name and the library name don't match, Python will raise an error when trying to import the module:

ImportError: dynamic module does not define 
    module export function (PyInit_name_of_your_module)

The name argument

You can also specify the name of the module explicitly using the name argument, rather than relying on the name of the annotated function:

#![allow(unused)]
fn main() {
#[pymodule]
#[pyo3(name = "setup")]
fn random_name(m: &Bound<'_, PyModule>) -> PyResult<()> {
   // [...]
}
}

Mysterious types?

You might be wondering: what's up with &Bound<'_, PyModule>? What about PyResult?
Don't worry, we'll cover these types in due time later in the course. Go with the flow for now!

Exercise

The exercise for this section is located in 01_intro/02_modules

Functions

Empty modules are not that useful: let's add some functions to our extension!
As you've seen in the "Setup" section, pyo3 provides another procedural macro to define functions that can be called from Python: #[pyfunction].

Back then we used it to define the it_works function:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

// 👇 A Python function defined in Rust
#[pyfunction]
fn it_works() -> bool {
    true
}
}

Unlike modules, functions aren't exposed to Python automatically; you must attach them to a module using the wrap_pyfunction! macro:

#![allow(unused)]
fn main() {
#[pymodule]
fn setup(m: &Bound<'_, PyModule>) -> PyResult<()> {
    // 👇 Expose the function to Python
    m.add_function(wrap_pyfunction!(it_works, m)?)?;
    Ok(())
}
}

Exercise

The exercise for this section is located in 01_intro/03_functions

Arguments

no_op, the function you added to solve the previous exercise, is very simple:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
    
#[pyfunction]
fn no_op() {
    // Do nothing
}
}

Let's take it up a notch: what if you want to pass a value from Python to Rust?

The FromPyObject trait

#[pyfunction] functions can take arguments, just like regular Rust functions.
But there's a catch: it must be possible to build those arguments from Python objects.

The contract is encoded in the FromPyObject trait, defined in pyo3:

#![allow(unused)]
fn main() {
pub trait FromPyObject<'py>: Sized {
    fn extract_bound(ob: &Bound<'py, PyAny>) -> PyResult<Self>;
}
}

We won't go into the details of FromPyObject's definition just yet: it would require an in-depth discussion of Python's Global Interpreter Lock (GIL) and the way pyo3 models it in Rust. We'll get to it in the next section.
For the time being, let's focus on what the trait unlocks for us: the ability to convert Python objects into Rust types.

Available implementations

pyo3 provides implementations of FromPyObject for a large number of types—e.g. i32, f64, String, Vec, etc. You can find an exhaustive list in pyo3's guide, under the "Rust" table column.

Conversion cost

Going from a Python object to a Rust type is not free—e.g. the in-memory representation of a Python list doesn't match the in-memory representation of a Rust Vec.
The conversion introduces a (usually small) overhead that you'll have to incur every time you invoke your Rust function from Python. It's a good trade-off if you end up performing enough computational work in Rust to amortize the conversion cost.

Python-native types

In pyo3's documentation you can see a column of "Python-native" types.
Don't try to use them to solve the exercise for this section: we'll cover them in the next one.

References

Exercise

The exercise for this section is located in 01_intro/04_arguments

Global Interpreter Lock (GIL)

If you go back to pyo3's documentation on arguments, you'll find a table column listing so called "Python-native" types. What are they, and why would you use them?

Python-native types

There is overhead in converting a Python object into a Rust-native type.
That overhead might dominate the cost of invoking your Rust function if the function itself isn't doing much computational work. In those cases, it can be desirable to work directly using Python's in-memory representation of the object. That's where the Py* types come in: they give you direct access to Python objects, with minimal overhead1.

Out of the entire family of Py* types, PyAny deserves a special mention. It's the most general Python-native type in pyo3: it stands for an arbitrary Python object. You can use it whenever you don't know the exact type of the object you're working with, or you don't care about it.

Py* types don't implement FromPyObject

Let's try to rewrite the solution of the previous exercise using PyList rather than Vec<u64>:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

fn print_number_list(list: &PyList) {
    todo!()
}
}

If you try to compile this code, you'll get an error:

error[E0277]: the trait bound `&PyList: PyFunctionArgument<'_, '_>` is not satisfied
   --> src/lib.rs:7:28
    |
7   | fn print_number_list(list: &PyList) {
    |                            ^ 
    |        the trait `PyClass` is not implemented for `&PyList`, 
    |        which is required by `&PyList: PyFunctionArgument<'_, '_>`
    |
    = help: the following other types implement trait `PyFunctionArgument<'a, 'py>`:
              &'a pyo3::Bound<'py, T>
              Option<&'a pyo3::Bound<'py, T>>
    = note: required for `&PyList` to implement `FromPyObject<'_>`
    = note: required for `&PyList` to implement `FromPyObjectBound<'_, '_>`
    = note: required for `&PyList` to implement `PyFunctionArgument<'_, '_>`

The error message is a bit cryptic because it mentions a number of private pyo3 traits (PyFunctionArgument and FromPyObjectBound), but the gist of it is that &PyList doesn't implement FromPyObject. That's true for all Py* types.

Confusing, isn't it? How is possible that Python-native types, that require no conversion, don't implement the trait that allows you to convert Python objects into Rust types?

It's time to have that talk, the one about Python's Global Interpreter Lock (GIL).

Global Interpreter Lock (GIL)

Out of the box, Python's2 data structures are not thread-safe. To prevent data races, there is a global mutual exclusion lock that allows only one thread to execute Python bytecode at a time—i.e. the so-called Global Interpreter Lock (GIL).

It is forbidden to interact with Python objects without holding the GIL.

That's why pyo3 doesn't implement FromPyObject for Py* types: it would allow you to interact with Python objects without you necessarily holding the GIL, a recipe for disaster.

Python<'py>

pyo3 uses a combination of lifetimes and smart pointers to ensure that you're interacting with Python objects in a safe way.

Python<'py> is the cornerstone of the entire system: it's a token type that guarantees that you're holding the GIL. All APIs that require you to hold the GIL will, either directly or indirectly, require you to provide a Python<'py> token as proof.

pyo3 will automatically acquire the GIL behind the scenes whenever you invoke a Rust function from Python. In fact, you can ask for a Python<'py> token as an argument to your Rust function, and pyo3 will provide it for you—it has no (additional) cost.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
// There is no runtime difference between invoking the two functions
// below from Python.
// The first one is just more explicit about the fact that it requires
// the caller to acquire the GIL ahead of time.

#[pyfunction]
fn print_number_list(_py: Python<'_>, list: Vec<u64>) {
    todo!()
}

#[pyfunction]
fn print_number_list2(list: Vec<u64>) {
    todo!()
}
}

'py, the lifetime parameter of Python<'py>, is used to represent how long the GIL is going to be held.

Bound<'py>

You won't be interacting with Python<'py> directly most of the time.
Instead, you'll use the Bound<'py, T> type, a smart pointer that encapsulates a reference to a Python object, ensuring that you're holding the GIL when you're interacting with it.

Using Bound<'py, T> we can finally start using the Py* types as function arguments:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyfunction]
fn print_number_list(list: Bound<'_, PyList>) {
    todo!()
}
}

Bound ensures that we're holding the GIL when interacting with the list instance that has been passed to us as function argument.

FromPyObject

We can now go back to the definition of the FromPyObject trait:

#![allow(unused)]
fn main() {
pub trait FromPyObject<'py>: Sized {
    fn extract_bound(ob: &Bound<'py, PyAny>) -> PyResult<Self>;
}
}

extract_bound takes a &Bound<'py, PyAny> as argument, rather than a bare &PyAny, to ensure that we're holding the GIL when we're interacting with the Python object during the conversion.

References

Footnotes

1

pyo3 still needs to ensure that the Python object you're working with is of the expected type. It'll therefore perform an isinstance check before handing you the object—e.g. checking that an object is indeed a list before giving you a PyList argument. The only exception to this rule is PyAny, which can represent an arbitrary Python object.

2

CPython is the reference implementation of Python, written in C. It's the most widely used Python interpreter and what most people refer to when they say "Python".

Exercise

The exercise for this section is located in 01_intro/05_gil

Output values

We've gone deep into the weeds of how pyo3 handles arguments to your #[pyfunction]s. Let's now move our focus to output values: how do you return something from your Rust functions to Python?

IntoPy

Guess what? There's a trait for that too!
IntoPy is the counterpart of FromPyObject. It converts Rust values into Python objects:

#![allow(unused)]
fn main() {
pub trait IntoPy<T>: Sized {
    fn into_py(self, py: Python<'_>) -> T;
}
}

The output type of your #[pyfunction] must implement IntoPy.

IntoPy::into_py

IntoPy::into_py expects two arguments:

  • self: the Rust value you want to convert into a Python object.
  • Python<'_>: a GIL token that you can use to create new Python objects.

Case study: a newtype

Let's look at a simple example: a newtype that wraps a u64. We want it to be represented as a "plain" integer in Python.

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

struct MyType {
    value: u64,
}

impl IntoPy<Py<PyAny>> for MyType {
    fn into_py(self, py: Python<'_>) -> Py<PyAny> {
        self.value.to_object(py)
    }
}
}

Provided implementations

pyo3 provides out-of-the-box implementations of IntoPy for many Rust types, as well as for all Py* types. Check out its documentation for an exhaustive list.

Exercise

The exercise for this section is located in 01_intro/06_output

Exceptions

Python and Rust have different error handling mechanisms.
In Python, you raise exceptions to signal that something went wrong.
In Rust, errors are normal values that you return from functions, usually via the Result type.

pyo3 provides PyResult<T> to help you bridge the gap between these two worlds.

PyResult<T>

PyResult<T> is the type you'll return whenever your #[pyfunction] can fail.
It is a type alias for Result<T, PyErr>, where PyErr is pyo3's representation of a Python exception. pyo3 will automatically raise a Python exception whenever a #[pyfunction] returns Err(PyErr) value:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::types::PyAny;

#[pyfunction]
fn print_if_number(item: Bound<'_, PyAny>) -> PyResult<()> {
    let number = item.extract::<u64>()?;
    println!("{}", number);
    Ok(())
}
}

In the example above, extract::<u64>()? returns a PyResult<u64>.
If the object is not an unsigned integer, extract will return an error, which will be propagated up to the caller via the ? operator. On the Python side, this error will be raised as a Python exception by pyo3.

Built-in exception types

You should be intentional about the types of exceptions you raise. What kind of error are you signaling? What is the caller expected to catch?

All built-in Python exceptions are available in pyo3::exceptions—e.g. pyo3::exceptions::PyValueError for a ValueError. You can use their new_err method to create an instance.

Panics

Rust provides another mechanism for handling "unrecoverable" errors: panics. What happens if you panic in a #[pyfunction]?
pyo3 will catch the panic and raise a pyo3_runtime.PanicException to the Python caller. You've probably seen this behaviour at play when solving the exercises associated to the previous sections.

Exercise

The exercise for this section is located in 01_intro/07_exceptions

Wrapping up

We've covered most of pyo3's key concepts in this chapter.
Before moving on, let's go through one last exercise to consolidate what we've learned. You'll have minimal guidance this time—just the exercise description and the tests to guide you.

Exercise

The exercise for this section is located in 01_intro/08_outro

Classes

We've covered Python functions written in Rust, but what about classes?

Defining a class

You can use the #[pyclass] attribute to define a new Python class in Rust. Here's an example:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct Wallet {
    balance: i32,
}
}

It defines a new Python class called Wallet with a single field, balance.

Registering a class

Just like with #[pyfunction]s, you must explicitly register your class with a module to make it visible to users of your extension.
Continuing with the example above, you'd register the Wallet class like this:

#![allow(unused)]
fn main() {
#[pymodule]
fn my_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_class::<Wallet>()?;
    Ok(())
}
}

IntoPy

Rust types that have been annotated with #[pyclass] automatically implement the IntoPy trait, thus allowing you to return them from your #[pyfunction]s.

For example, you can define a function that creates a new Wallet instance:

#![allow(unused)]
fn main() {
#[pyfunction]
fn new_wallet(balance: i32) -> Wallet {
    Wallet { balance }
}
}

It'll compile just fine, handing over a new Wallet instance to the Python caller.

Attributes

By default, the fields of your #[pyclass]-annotated structs aren't accessible to Python callers.
Going back to our Wallet example—if you try to access the balance field from Python, you'll get an error:

        wallet = new_wallet(0)
>       assert wallet.balance == 0
E       AttributeError: 'builtins.Wallet' object has no attribute 'balance'

tests/test_sample.py:8: AttributeError

The same error would occur even if you made balance a public field.

To make the field accessible to Python, you must add a getter.
This can be done using the #[pyo3(get)] attribute:

#![allow(unused)]
fn main() {
#[pyclass]
struct Wallet {
    #[pyo3(get)]
    balance: i32,
}
}

Now, the balance field is accessible from Python:

def test_wallet():
    wallet = new_wallet(0)
    assert wallet.balance == 0

If you want to allow Python callers to modify the field, you can add a setter using the #[pyo3(set)] attribute:

#![allow(unused)]
fn main() {
#[pyclass]
struct Wallet {
    // Both getter and setter
    #[pyo3(get, set)]
    balance: i32,
}
}

Exercise

The exercise for this section is located in 02_classes/00_pyclass

Constructors

In the previous section (and its exercise) we relied on a #[pyfunction] as the constructor for the #[pyclass] we defined. Without new_wallet, we wouldn't have been able to create new Wallet instances from Python.
Let's now explore how to define a constructor directly within the #[pyclass] itself.

Defining a constructor

You can add a constructor to your #[pyclass] using the #[new] attribute on a method. Here's an example:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct Wallet {
    #[pyo3(get, set)]
    balance: i32,
}

#[pymethods]
impl Wallet {
    #[new]
    fn new(balance: i32) -> Self {
        Wallet { balance }
    }
}
}

A Rust method annotated with #[new] is equivalent to the __new__ method in Python. At the moment there is no way to define the __init__ method in Rust.
The impl block containing the constructor must also be annotated with the #[pymethods] attribute for #[new] to work as expected.

Signature

Everything we learned about arguments in the context of #[pyfunction]s applies to constructors as well.
In terms of output type, you can return Self if the constructor is infallible, or PyResult<Self> if it can fail.

Exercise

The exercise for this section is located in 02_classes/01_constructors

Methods

The #[pymethods] attribute is not limited to constructors. You can use it to attach any number of methods to your #[pyclass]:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct Wallet {
    #[pyo3(get, set)]
    balance: i32,
}

#[pymethods]
impl Wallet {
    #[new]
    fn new(balance: i32) -> Self {
        Wallet { balance }
    }

    fn deposit(&mut self, amount: i32) {
        self.balance += amount;
    }

    fn withdraw(&mut self, amount: i32) {
        self.balance -= amount;
    }
}
}

All methods within an impl block annotated with #[pymethods] are automatically exposed to Python as methods on your #[pyclass]1. The deposit and withdraw methods in the example above can be called from Python like this:

wallet = Wallet(0)
wallet.deposit(100)
wallet.withdraw(50)
assert wallet.balance == 50

multiple-pymethods

You can't annotate multiple impl blocks with #[pymethods] for the same class, due to a limitation in Rust's metaprogramming capabilities.
There is a way to work around this issue using some linker dark magic, via the multiple-pymethods feature flag, but it comes with a penalty in terms of compile times as well as limited cross-platform support. Check out pyo3's documentation for more details.

Footnotes

1

All methods in a #[pymethods] block are exposed, even if they are private!

Exercise

The exercise for this section is located in 02_classes/02_methods

Custom setters and getters

In a previous section, we learned how to attach the default getter and setter to a field in a #[pyclass]:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct Wallet {
    #[pyo3(get, set)]
    balance: i32,
}
}

This is convenient, but it's not always desirable!
Let's introduce an additional constraint to our Wallet struct: the balance should never go below a pre-determined overdraft threshold. We'd start by enforcing this constraint in the constructor method:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;
use pyo3::exceptions::PyValueError;

#[pyclass]
struct Wallet {
    #[pyo3(get, set)]
    balance: i32,
}

const OVERDRAFT_LIMIT: i32 = -100;

#[pymethods]
impl Wallet {
    #[new]
    fn new(balance: i32) -> PyResult<Self> {
        if balance < OVERDRAFT_LIMIT {
           return Err(PyValueError::new_err("Balance cannot be below overdraft limit"));     
        }
        Ok(Wallet { balance })
    }
}
}

Wallet::new ensures that a newly-created Wallet upholds the overdraft constraint. But the default setter can be easily used to circumvent the limit:

wallet = Wallet(0)
wallet.balance = -200 # This should not be allowed, but it is!

#[setter] and #[getter]

We can override the default getter and setter by defining custom methods for them.
Here's how we can implement a custom setter for the balance field via the #[setter] attribute:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct Wallet {
    // We keep using the default getter, no issues there
    #[pyo3(get)]
    balance: i32,
}

const OVERDRAFT_LIMIT: i32 = -100;

#[pymethods]
impl Wallet {
    #[new]
    fn new(balance: i32) -> PyResult<Self> {
        Wallet::check_balance(balance)?;
        Ok(Wallet { balance })
    }

    #[setter]
    fn set_balance(&mut self, value: i32) {
        Wallet::check_balance(value)?;
        self.balance = value;
    }
}

impl Wallet {
    // We put this method in a separate `impl` block to avoid exposing it to Python
    fn check_balance(balance: i32) -> PyResult<()> {
        if balance < OVERDRAFT_LIMIT {
            return Err(PyValueError::new_err("Balance cannot be below overdraft limit"));
        }
        Ok(())
    }
}
}

Every time the balance field is set in Python, Wallet::set_balance will be called:

wallet = Wallet(0)
wallet.balance = -200  # Now raises a `ValueError`

The field is associated with its setter using a conventional naming strategy for the setter method: set_<field_name>. You can also explicitly specify the field name in the #[setter] attribute, like this: #[setter(balance)].

Custom getters are defined in a similar way using the #[getter] attribute. The naming convention for getter methods is <field_name>, but you can also specify the field name explicitly in the attribute—e.g. #[getter(balance)].

Exercise

The exercise for this section is located in 02_classes/03_setters

Static methods

All the class methods we've seen so far have been instance methods—i.e. they take an instance of the class as one of their arguments.
Python supports static methods as well. These methods don't take an instance of the class as an argument, but they are "attached" to the class itself.

The same concept exists in Rust:

#![allow(unused)]
fn main() {
pub struct Wallet {
    balance: i32,
}

impl Wallet {
    pub fn default() -> Self {
        Wallet { balance: 0 }
    }
}
}

Wallet::get_default is a static method since it doesn't take self or references to self as arguments.
You might then expect the following to define a Python static method on the Wallet class:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct Wallet {
    #[pyo3(get, set)]
    balance: i32,
}

#[pymethods]
impl Wallet {
    #[new]
    fn new(balance: i32) -> Self {
        Wallet { balance }
    }

    fn default() -> Self {
        Wallet { balance: 0 }
    }
}
}

However, this code will not compile.
To define a static method in Python, you need to explicitly mark it with the #[staticmethod] attribute:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct Wallet {
    #[pyo3(get, set)]
    balance: i32,
}

#[pymethods]
impl Wallet {
    #[new]
    fn new(balance: i32) -> Self {
        Wallet { balance }
    }
 
    // Notice the `#[staticmethod]` attribute here!
    #[staticmethod]
    fn default() -> Self {
        Wallet { balance: 0 }
    }
}
}

Class methods

Python also supports class methods. These methods take the class itself as an argument, rather than an instance of the class.
In Rust, you can define class methods by taking cls: &PyType as the first argument:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass]
struct Wallet {
    #[pyo3(get, set)]
    balance: i32,
}

#[pymethods]
impl Wallet {
    #[new]
    fn new(balance: i32) -> Self {
        Wallet { balance }
    }

    // Notice the `cls` argument here!
    #[classmethod]
    fn from_str(_cls: &Bound<'_, PyType>, balance: &str) -> PyResult<Self> {
        let balance = balance.parse::<i32>()?;
        Ok(Wallet { balance })
    }
}
}

Since you can directly refer to the class in a Rust static method (i.e. the Self type), you won't find yourself using class methods as often as you would in Python.

Exercise

The exercise for this section is located in 02_classes/04_static_methods

Inheritance

Python, unlike Rust, supports inheritance.
Each class in Python can inherit attributes and methods from a parent class.

class Parent:
    def __init__(self, name):
        self.name = name

    def greet(self):
        print(f"Hello, {self.name}!")
        
# Declare `Child` as a subclass of `Parent`
class Child(Parent):
    def __init__(self, name, age):
        # Call the parent class's constructor
        super().__init__(name)
        self.age = age
        
child = Child("Alice", 7)
# `Child` inherits the `greet` method from `Parent`, so we can call it
child.greet() # Prints "Hello, Alice!"

pyo3 and inheritance

pyo3 supports inheritance as well, via additional attributes on the #[pyclass] macro.
To understand how it works, let's try to translate the Python example above to Rust. We'll start with defining the base class, Parent:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass(subclass)]
struct Parent {
    name: String,
}

#[pymethods]
impl Parent {
    #[new]
    fn new(name: String) -> Self {
        Parent { name }
    }

    fn greet(&self) {
        println!("Hello, {}!", self.name);
    }
}
}

You can spot one new attribute in the #[pyclass] macro: subclass. This attribute tells pyo3 that this class can be subclassed, and it should generate the necessary machinery to support inheritance.

Now let's define the Child class, which inherits from Parent:

#![allow(unused)]
fn main() {
#[pyclass(extends=Parent)]
struct Child {
    age: u8,
}
}

We're using the extends attribute to specify that Child is a subclass of Parent.
Things get a bit more complicated when it comes to the constructor:

#![allow(unused)]
fn main() {
#[pymethods]
impl Child {
    #[new]
    fn new(name: String, age: u8) -> PyClassInitializer<Self> {
        let parent = Parent::new(name);
        let child = Self { age };
        PyClassInitializer::from(parent).add_subclass(child)
    }
}
}

Whenever you initialize a subclass, you need to make sure that the parent class is initialized first.
We start by calling Parent::new to create an instance of the parent class. We then initialize Child, via Self { age }. We then use PyClassInitializer to return both the parent and child instances together.

Even though Child doesn't have a greet method on the Rust side, you'll be able to call it from Python since the generated Child class inherits it from Parent.

Nested inheritance

PyClassInitializer can be used to build arbitrarily deep inheritance hierarchies. For example, if Child had its own subclass, you could call add_subclass again to add yet another subclass to the chain.

#![allow(unused)]
fn main() {
#[pyclass(extends=Child)]
struct Grandchild {
    hobby: String,
}

#[pymethods]
impl Grandchild {
    #[new]
    fn new(name: String, age: u8, hobby: String) -> PyClassInitializer<Self> {
        let child = Child::new(name, age);
        let grandchild = Self { hobby };
        PyClassInitializer::from(child).add_subclass(grandchild)
    }
}
}

Limitations

pyo3 supports two kinds of superclasses:

  • A Python class defined in Rust, via #[pyclass]
  • A Python built-in class, like PyDict or PyList

It currently doesn't support using a custom Python class as the parent class for a class defined in Rust.

Exercise

The exercise for this section is located in 02_classes/05_inheritance

Parent class

Let's go back to our example from the previous section:

#![allow(unused)]
fn main() {
use pyo3::prelude::*;

#[pyclass(subclass)]
struct Parent {
    name: String,
}

#[pymethods]
impl Parent {
    #[new]
    fn new(name: String) -> Self {
        // [...]
    }

    fn greet(&self) {
        println!("Hello, {}!", self.name);
    }
}

#[pyclass(extends=Parent)]
struct Child {
    age: u8,
}

#[pymethods]
impl Child {
    #[new]
    fn new(name: String, age: u8) -> PyClassInitializer<Self> {
        // [...]
    }
}
}

Child.greet is not defined, therefore it falls back to the Parent.greet method.
What if we wanted to override it in Child?

Overriding methods

On the surface, it's simple: just define a method with the same name in the subclass.

#![allow(unused)]
fn main() {
#[pymethods]
impl Child {
    #[new]
    fn new(name: String, age: u8) -> PyClassInitializer<Self> {
        // [...]
    }
    
    fn greet(&self) {
        println!("Hi, I'm {} and I'm {} years old!", self.name, self.age);
    }
}
}

There's an issue though: self.name won't work because the Rust struct for Child doesn't have a name field. At the same time, the Python Child class does, because it inherits it from Parent.

How do we fix this?

as_super to the rescue

We need a way, in Rust, to access the fields and methods of the parent class from the child class.
This can be done using another one of pyo3's smart pointers: PyRef.

#![allow(unused)]
fn main() {
#[pymethods]
impl Child {
    // [...]
    
    fn greet(self_: PyRef<'_, Self>) {
        todo!()
    }
}
}

PyRef represents an immutable reference to the Python object.
It allows us, in particular, to call the as_super method, which returns a reference to the parent class.

#![allow(unused)]
fn main() {
#[pymethods]
impl Child {
    // [...]
    
    fn greet(self_: PyRef<'_, Self>) {
        // This is now a reference to a `Parent` instance!
        let parent = self_.as_super();
        println!("Hi, I'm {} and I'm {} years old!", parent.name, self_.age);
    }
}
}

Now we can access the name field from the parent class, and the age field from the child class.

PyRef and PyRefMut

PyRef is for immutable references, but what if we need to modify the parent class?
In that case, we can use PyRefMut, which is a mutable reference.

Exercise

The exercise for this section is located in 02_classes/06_parent

Wrapping up

There's a ton of little details and options when it comes to writing Python classes in Rust. We've covered the key concepts and most common use cases, but make sure to check out the official pyo3 documentation whenever you need more information about a specific feature (e.g. how do I declare a class to be frozen? How do I make my class iterable?).

Let's take a moment to reflect on what we've learned so far with one last exercise.

Exercise

The exercise for this section is located in 02_classes/07_outro