Python Code Generation Internals

This chapter explains how ros-z generates Python bindings for ROS 2 messages, including the architecture, generated code structure, and design rationale.

Architecture Overview

ros-z uses a hybrid approach: Python classes for ergonomics, Rust for serialization performance. The architecture uses derive macros for automatic Python-Rust conversion of arbitrarily nested message types.

graph TB
    subgraph "Build Time (cargo build)"
        ROS[".msg/.srv files"] --> Parser["ros-z-codegen<br/>Parser"]
        Parser --> Resolver["Dependency<br/>Resolver"]
        Resolver --> PyGen["Python Generator"]
        Resolver --> RsGen["Rust Generator"]
        PyGen --> PyFiles["*.py files<br/>(msgspec structs)"]
        RsGen --> RsFile["Rust structs with<br/>derive macros"]
        RsGen --> PyO3File["python_bindings.rs<br/>(serialize/deserialize)"]
    end

    subgraph "Derive Macros (ros-z-derive)"
        Derive["FromPyMessage<br/>IntoPyMessage"]
        Derive -->|"generates"| Extract["Field extraction code"]
        Derive -->|"generates"| Construct["Object construction code"]
    end

    subgraph "Runtime"
        PyFiles --> PyObj["Python Object"]
        PyObj -->|"FromPyMessage::from_py()"| RsStruct["Rust Struct"]
        RsStruct -->|"ros-z-cdr::to_vec()"| CDR["CDR Bytes"]
        CDR -->|"Zenoh"| Network["Network"]
    end

Key Components

Generated Code Examples

Python: msgspec Structs

For each ROS message, a frozen msgspec.Struct is generated:

# Generated from std_msgs/msg/String.msg
class String(msgspec.Struct, frozen=True, kw_only=True):
    data: str = ""

    __msgtype__: ClassVar[str] = 'std_msgs/msg/String'
    __hash__: ClassVar[str] = 'RIHS01_df668c740482bbd48fb39d76a70dfd4bd59db1288021743503259e948f6b1a18'

Nested messages use forward references:

# Generated from geometry_msgs/msg/Twist.msg
class Twist(msgspec.Struct, frozen=True, kw_only=True):
    linear: "geometry_msgs.Vector3 | None" = None
    angular: "geometry_msgs.Vector3 | None" = None

    __msgtype__: ClassVar[str] = 'geometry_msgs/msg/Twist'
    __hash__: ClassVar[str] = 'RIHS01_9c45bf16fe0983d80e3cfe750d6835843d265a9a6c46bd2e609fcddde6fb8d2a'

Service request/response types use the service type hash in __hash__:

# Generated from example_interfaces/srv/AddTwoInts.srv
class AddTwoIntsRequest(msgspec.Struct, frozen=True, kw_only=True):
    a: int = 0
    b: int = 0

    __msgtype__: ClassVar[str] = 'example_interfaces/msg/AddTwoIntsRequest'
    __hash__: ClassVar[str] = 'RIHS01_e118de6bf5eeb66a2491b5bda11202e7b68f198d6f67922cf30364858239c81a'

class AddTwoIntsResponse(msgspec.Struct, frozen=True, kw_only=True):
    sum: int = 0

    __msgtype__: ClassVar[str] = 'example_interfaces/msg/AddTwoIntsResponse'
    __hash__: ClassVar[str] = 'RIHS01_e118de6bf5eeb66a2491b5bda11202e7b68f198d6f67922cf30364858239c81a'

For service types, __hash__ contains the service type hash (computed from the combined request/response definition). Both request and response share the same hash since they belong to the same service. This differs from regular messages where __hash__ contains the individual message type hash.

Rust: Generated Structs with Derive Macros

The Rust code generator adds derive attributes to message structs:

// Generated from std_msgs/msg/String.msg
#[derive(Debug, Clone, Default, ::serde::Serialize, ::serde::Deserialize)]
#[cfg_attr(feature = "python_registry", derive(::ros_z_derive::FromPyMessage, ::ros_z_derive::IntoPyMessage))]
#[cfg_attr(feature = "python_registry", ros_msg(module = "ros_z_msgs_py.types.std_msgs"))]
pub struct String {
    pub data: std::string::String,
}

For nested messages with optional fields:

// Generated from geometry_msgs/msg/Twist.msg
#[derive(Debug, Clone, Default, ::serde::Serialize, ::serde::Deserialize)]
#[cfg_attr(feature = "python_registry", derive(::ros_z_derive::FromPyMessage, ::ros_z_derive::IntoPyMessage))]
#[cfg_attr(feature = "python_registry", ros_msg(module = "ros_z_msgs_py.types.geometry_msgs"))]
pub struct Twist {
    pub linear: Vector3,
    pub angular: Vector3,
}

Derive Macros: Zero-Copy Byte Array Handling

Fields annotated with #[ros_msg(zbuf)] (applied automatically by the code generator for uint8[]/byte[] fields) get special treatment in the derive macros:

FromPyMessage (Python → Rust) tries three paths in order:

1. ZBufView — extracts the inner ZBuf via clone() (reference-counted, zero-copy)
2. bytes/bytearray — copies into a new ZBuf
3. list[int] — fallback extraction

// Generated by FromPyMessage derive macro for #[ros_msg(zbuf)] fields
let py_attr = obj.getattr("data")?;
if let Ok(view) = py_attr.downcast::<ZBufView>() {
    view.borrow().zbuf().clone()     // Arc clone — no data copy
} else if let Ok(bytes) = py_attr.downcast::<PyBytes>() {
    ZBuf::from(bytes.as_bytes().to_vec())
} else if let Ok(bytearray) = py_attr.downcast::<PyByteArray>() {
    ZBuf::from(unsafe { bytearray.as_bytes() }.to_vec())
} else {
    let bytes: Vec<u8> = py_attr.extract()?;
    ZBuf::from(bytes)
}

IntoPyMessage (Rust → Python) creates a ZBufView instead of copying into PyBytes:

// Generated by IntoPyMessage derive macro for #[ros_msg(zbuf)] fields
let zbuf_view = ZBufView::new(self.data.clone());
let py_view = Py::new(py, zbuf_view)?;
kwargs.set_item("data", py_view)?;

This means a received message's byte array fields can be passed directly to a new message for re-publishing without any data copy.

Rust: PyO3 Serialize/Deserialize Functions

The serialize/deserialize functions use the derive macro traits:

// Generated serialize function - uses FromPyMessage trait
#[pyfunction]
pub fn serialize_string(py: Python, msg: &Bound<'_, PyAny>) -> PyResult<Vec<u8>> {
    use ::pyo3::types::PyAnyMethods;
    use ::ros_z::python_bridge::FromPyMessage;

    // Derive macro handles all field extraction recursively
    let rust_msg = <ros::std_msgs::String>::from_py(msg)?;

    // CDR serialization with 4-byte encapsulation header
    let mut cdr_data = ros_z_cdr::to_vec::<_, ros_z_cdr::LittleEndian>(&rust_msg, 256)
        .map_err(|e| pyo3::exceptions::PyValueError::new_err(e.to_string()))?;
    let mut result = vec![0x00, 0x01, 0x00, 0x00]; // CDR encapsulation header
    result.append(&mut cdr_data);
    Ok(result)
}

// Generated deserialize function - uses IntoPyMessage trait
#[pyfunction]
pub fn deserialize_string(py: Python, bytes: &[u8]) -> PyResult<PyObject> {
    use ::ros_z::python_bridge::IntoPyMessage;

    // Skip 4-byte CDR encapsulation header
    let cdr_data = &bytes[4..];
    let (rust_msg, _): (ros::std_msgs::String, _) = ros_z_cdr::from_bytes::<_, ros_z_cdr::LittleEndian>(cdr_data)
        .map_err(|e| pyo3::exceptions::PyValueError::new_err(e.to_string()))?;

    // Derive macro handles all field construction recursively
    rust_msg.into_py_message(py)
}

Data Flow

Publishing (Python to Wire)

sequenceDiagram
    participant App as Python App
    participant Msg as msgspec.Struct
    participant PyO3 as PyO3 Layer
    participant Rust as Rust Struct
    participant CDR as CDR Encoder
    participant Net as Zenoh Network

    App->>Msg: Create message
    App->>PyO3: publish(msg)
    PyO3->>PyO3: getattr() each field
    PyO3->>Rust: Construct Rust struct
    Rust->>CDR: cdr::serialize()
    CDR->>Net: Send bytes

Subscribing (Wire to Python)

sequenceDiagram
    participant Net as Zenoh Network
    participant CDR as CDR Decoder
    participant Rust as Rust Struct
    participant PyO3 as PyO3 Layer
    participant Msg as msgspec.Struct
    participant App as Python App

    Net->>CDR: Receive bytes
    Note over CDR: Store ZBuf in thread-local<br/>for zero-copy deser
    CDR->>Rust: cdr::deserialize()
    Note over CDR,Rust: byte[] fields → ZSlice sub-view<br/>(zero-copy)
    Rust->>PyO3: Build PyDict kwargs
    Note over PyO3: byte[] → ZBufView<br/>(buffer protocol)
    PyO3->>Msg: Call class(**kwargs)
    Msg->>App: Return message

Type Mapping

ROS to Python Type Mapping

Design: Hybrid Python + Rust with Derive Macros

┌─────────────────────────────────────────────────────────────┐
│                    Current Architecture                      │
├─────────────────────────────────────────────────────────────┤
│  Python Layer          │  Rust Layer (via PyO3)             │
│  ─────────────         │  ──────────────────────            │
│  • msgspec structs     │  • Derive macros                   │
│  • Type annotations    │  • Trait-based conversion          │
│  • IDE support         │  • CDR serialization               │
│  • User-facing API     │  • Zenoh transport                 │
│                        │  • Memory management               │
└─────────────────────────────────────────────────────────────┘

Features:

• Fast serialization (Rust CDR implementation)
• Type-safe Python classes with IDE support
• Immutable messages (frozen=True)
• Automatic type hash validation
• Zero-copy byte array handling: ZBufView exposes received data via buffer protocol without copying; FromPyMessage detects ZBufView inputs and extracts the inner buffer via reference counting
• Derive macros handle arbitrary nesting depth automatically
• None handling for optional nested fields
• Minimal generated code (traits do the work)

Build Process

Step-by-step Flow

flowchart TD
    A["cargo build ros-z-msgs<br/>--features python_registry"] --> B["build.rs executes"]
    B --> C["Discover ROS packages<br/>(AMENT_PREFIX_PATH or bundled)"]
    C --> D["Parse .msg/.srv files"]
    D --> E["Resolve dependencies<br/>(type hashes, nested types)"]
    E --> F["Generate Python files"]
    E --> G["Generate Rust PyO3 module"]
    F --> H["ros-z-msgs/python/<br/>ros_z_msgs_py/types/*.py"]
    G --> I["$OUT_DIR/python_bindings.rs"]
    I --> J["include!() in lib.rs"]
    J --> K["Compiled into ros-z-msgs"]

Output Locations

Feature Flags

The Python codegen is gated behind the python_registry feature:

#![allow(unused)]
fn main() {
// lib.rs
#[cfg(feature = "python_registry")]
include!(concat!(env!("OUT_DIR"), "/python_bindings.rs"));
}

The ros-z-derive crate provides the derive macros, and ros-z/python enables the FromPyMessage and IntoPyMessage traits.