Using Yoke with a custom cart datatype wrapping a Vec

[G33KatWork]

Hi,

I am currently writing a protocol stack in Rust. For incoming packets, I built some zerocopy packet parsers based on Fuchsia's packet library, because I liked the ideas described in this presentation: https://www.youtube.com/watch?v=UfMOOxOGCmA

The way this works is that you wrap some sequence of bytes in a Buf type which in my case contains a Vec<u8> and a range where the current body of the packet to parse is. You can use any container as storage for your packet as long as it implements AsRef<[u8]>. It looks like this:

pub struct Buf<B> {
    buf: B,
    body: Range<usize>,
}

After creating a Buf with the data and a range of the data, you can call parse on it, because this Buf implements a trait called ParseBuffer. Doing that allows you to parse packets using the zerocopy crate. In the end it looks like this:

let v = vec![0x29, 0x03, 0x03, 0x01, 0x01, 0x55, 0xAA, 0x00, 0x17, 0x00, 0x0C, 0, 1, 2, 3, 3, 4, 5, 7, 8, 9];
let mut b = Buf::new(v, ..);

println!("buf before parsing: {:?}", b);
{
    let p = b.parse::<CemiPacket<&[u8], CemiLDataInd>>().unwrap();
    println!("parsed packet: {:?}", p);
}
println!("buf after parsing: {:?}", b);

Which yields this output:

buf before parsing: Buf { buf: [41, 3, 3, 1, 1, 85, 170, 0, 23, 0, 12, 0, 1, 2, 3, 3, 4, 5, 7, 8, 9], body: 0..21 }
parsed packet: CemiPacket { header: Ref(Header { prefix: HeaderPrefix { message_code: 41, additional_info_len: 3 } }), additional_info: [3, 1, 1], message: Ref(CemiLDataInd { ctrl1: Ctrl1Field(85), ctrl2: Ctrl2Field(170), addr_src: [0, 23], addr_dst: [0, 12] }), npdu: [0, 1, 2, 3, 3, 4, 5, 7, 8, 9] }
buf after parsing: Buf { buf: [41, 3, 3, 1, 1, 85, 170, 0, 23, 0, 12, 0, 1, 2, 3, 3, 4, 5, 7, 8, 9], body: 11..21 }

CemiPacket and CemiLDataInd are just a bunch of zerocopy ByteSlices and a few LayoutVerifieds that point to data in the Buf:

#[derive(Debug, Yokeable)]
pub struct CemiPacket<B: ByteSlice, M: CemiMessage> {
    header: LayoutVerified<B, Header>,
    additional_info: B,
    message: LayoutVerified<B, M>,
    npdu: B,
}

#[derive(Copy, Clone, Default, Debug, FromZeroes, FromBytes, AsBytes, Unaligned)]
#[repr(C)]
struct HeaderPrefix {
    message_code: u8,
    additional_info_len: u8,
}

#[derive(Copy, Clone, Debug, Eq, PartialEq, FromZeroes, FromBytes, AsBytes, Unaligned)]
#[repr(C)]
pub struct CemiLDataInd {
    ctrl1: Ctrl1Field,
    ctrl2: Ctrl2Field,
    addr_src: [u8; 2],
    addr_dst: [u8; 2],
}

The B generic type is where the lifetime of the buffer, this struct has been parsed from, is hidden. It is usually just a &[u8] with the lifetime that is the same as the one of the Buf. Rustc is perfectly capable to infer this, so most of the time when using the code it's just _. That's how I currently at least think is how all of this works.

So far, so good.

Now I want to pass around the parsed representation and the Buf that stores the data from one of my protocol layers to the next. This is exactly what Yoke was built for, but I can't manage to use it with the Buf type. One of the problems is that Buf needs to be mutable when parsing things, because the body range will be modified depending on how much bytes were consumed when parsing the payload. This is useful when you want to forward a packet from one machine to another as you can just undo the current parsing step by moving the body range and throwing it out again on another network interface.

I tried using Buf<Vec<u8>> directly as a cart like this:

let v = vec![0x29, 0x03, 0x03, 0x01, 0x01, 0x55, 0xAA, 0x00, 0x17, 0x00, 0x0C, 0, 1, 2, 3, 3, 4, 5, 7, 8, 9];

let a: Yoke<CemiPacket<&[u8], CemiLDataInd>, Buf<Vec<u8>>> = Yoke::attach_to_cart(Buf::new(v, ..), |mut x| {
    x.parse::<CemiPacket<&'static [u8], CemiLDataInd>>().unwrap()
});

At first Yoke complained that Deref isn't implemented for Buf, so I implemented this along with StableDeref for Buf<Vec<u8>> which should be fine.

But now the compiler complains that x doesn't outlive 'static and I can't figure out why. I also tried to replicate what Yoke does myself to have a little playground and figure out how all this works and I have the same or at least a similar problem:

pub struct MyFakeYoke<C: AsRef<[u8]>> {
    yokeable: KindaSortaDangling<CemiPacket<&'static [u8], CemiLDataInd>>,
    cart: Buf<C>,
}


fn test_packet_yoke() {
    let data = vec![0x29, 0x03, 0x03, 0x01, 0x01, 0x55, 0xAA, 0x00, 0x17, 0x00, 0x0C, 0, 1, 2, 3, 3, 4, 5, 7, 8, 9];
    let mut cart = Buf::new(data, ..);

    let p = cart.parse::<CemiPacket<&[u8], CemiLDataInd>>().unwrap();

    let yokeable = unsafe { <CemiPacket<&[u8], CemiLDataInd> as yoke::Yokeable>::make(p) };

    let foo: MyFakeYoke<Vec<u8>> = MyFakeYoke {
        cart,
        yokeable: KindaSortaDangling::new(yokeable)
    };
}

This is what rustc complains about:

error[E0597]: `cart` does not live long enough
   --> src/util/yoke/yoke.rs:122:17
    |
120 |         let mut cart = Buf::new(data, ..);
    |             -------- binding `cart` declared here
121 |
122 |         let p = cart.parse::<CemiPacket<&[u8], CemiLDataInd>>().unwrap();
    |                 ^^^^-------------------------------------------
    |                 |
    |                 borrowed value does not live long enough
    |                 argument requires that `cart` is borrowed for `'static`
...
151 |     }
    |     - `cart` dropped here while still borrowed

error[E0505]: cannot move out of `cart` because it is borrowed
   --> src/util/yoke/yoke.rs:127:13
    |
120 |         let mut cart = Buf::new(data, ..);
    |             -------- binding `cart` declared here
121 |
122 |         let p = cart.parse::<CemiPacket<&[u8], CemiLDataInd>>().unwrap();
    |                 -----------------------------------------------
    |                 |
    |                 borrow of `cart` occurs here
    |                 argument requires that `cart` is borrowed for `'static`
...
127 |             cart,
    |             ^^^^ move out of `cart` occurs here
    |
help: consider cloning the value if the performance cost is acceptable
    |
122 |         let p = cart.clone().parse::<CemiPacket<&[u8], CemiLDataInd>>().unwrap();
    |                     ++++++++

When trying to do the same with a simple custom struct with just two u8s and postcard::from_bytes, it works fine. And ParseBuffer::parse has a similar, if not the same, signature with regards to lifetimes as postcard::from_bytes:

fn parse<'a, P: ParsablePacket<&'a [u8], ()>>(&'a mut self) -> Result<P, P::Error>

I have to admit I think I get the concept on how all of this works and I have read your blog post multiple times to understand what's going on, but I am slightly overwhelmed by all the fine details and I don't know where else to ask.

A nudge in the right direction on how I can stuff one of these Bufs and a parsed packet into the same struct by using lifetime erasure would be extremely helpful. Preferably even by using Yoke itself, because you already did all the crazy complicated work to get this safe and sound which is way above my head right now.

[sffc]

The closure you pass to attach_to_cart should return something that borrows from the argument x. This looks wrong:

x.parse::<CemiPacket<&'static [u8], CemiLDataInd>>().unwrap()

The 'static there implies that you need to borrow x as 'static, but you should be able to borrow it for the short-lived function lifetime. Yoke takes care of the erasure after the attach_to_cart callback finishes.

[G33KatWork]

Yeah, you are right. That was a leftover from my countless experiments.

let v = vec![0x29, 0x03, 0x03, 0x01, 0x01, 0x55, 0xAA, 0x00, 0x17, 0x00, 0x0C, 0, 1, 2, 3, 3, 4, 5, 7, 8, 9];

let a: Yoke<CemiPacket<&[u8], CemiLDataInd>, Buf<Vec<u8>>> = Yoke::attach_to_cart(Buf::new(v, ..), |mut x| {
    x.parse::<CemiPacket<&[u8], CemiLDataInd>>().unwrap()
});

complains with

error: lifetime may not live long enough
  --> src/util/yoke/yoke.rs:79:13
   |
78 |         let a: Yoke<CemiPacket<&[u8], CemiLDataInd>, Buf<Vec<u8>>> = Yoke::attach_to_cart(Buf::new(v, ..), |mut x| {
   |                                                                                                             ----- has type `&'1 [u8]`
79 |             x.parse::<CemiPacket<&[u8], CemiLDataInd>>().unwrap()
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'static`

error[E0515]: cannot return value referencing function parameter `x`
  --> src/util/yoke/yoke.rs:79:13
   |
79 |             x.parse::<CemiPacket<&[u8], CemiLDataInd>>().unwrap()
   |             -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |             |
   |             returns a value referencing data owned by the current function
   |             `x` is borrowed here

The same happens when I just let the compiler infer the reference types.

I think it's my Deref implementation that messes up the lifetimes, because I call as_ref and return a reference to a slice:

impl<B: AsRef<[u8]>> Deref for Buf<B> {
    type Target = [u8];

    fn deref(&self) -> &Self::Target {
        &self.as_ref()[self.body.clone()]
    }
}

[sffc]

The signature of fn parse is super important. Is it this one you posted?

fn parse<'a, P: ParsablePacket<&'a [u8], ()>>(&'a mut self) -> Result<P, P::Error>

That would not work because you are borrowing x as an &mut, so any additional borrows borrow from the short-lived mutable reference. You can't mutate x inside of attach_to_cart.

[G33KatWork]

Yeah, this is correct. The problem is while parsing, the body range gets modified and keeps track of how much of the data in Buf is still available for parsing in the next packet layer. I feared that this would also be a problem.

Thank you very much for your help though! I think I'll try to make the whole packet parsing less complicated and scrap a few features from the whole packet library so that I can use simple Vecs that I don't need to mutate. Or I later implement a custom Yoke that keeps track of this body range outside of the parsing to get that feature back should I need it. It is mainly used because you can have footers and not only headers. I don't even need that in the protocol I try to implement a stack for. There are just no footers.

[sffc]

Although the attach_to_cart closure cannot mutate x, it can borrow other variables from the context and write out data to them, like how many bytes were consumed from the buffer, just like any other closure. It just cannot return data borrowing from anything other than x.

[G33KatWork]

I have to bother you again, because I am again stuck.

I tried redesigning my API a bit so that it works more like postcard, but I am having problems with the ByteSlice trait which hides the lifetime of deserialized data that falls out of the zerocopy crate.

This is my current, very crude, test case:

use zerocopy::ByteSlice;
use yoke::{Yoke, Yokeable};
use postcard::de_flavors::{Flavor, Slice};

#[derive(Debug, Yokeable)]
#[repr(C)]
struct TestStruct<B: ByteSlice> {
    a: B
}

fn yoke_test() {
    let v = vec![1, 2, 3, 4, 5];

    let a: Yoke<TestStruct<_>, Vec<u8>> = Yoke::attach_to_cart(v, |x| {
        let a = Slice::new(x).try_take_n(4).unwrap();
        let foo = TestStruct { a };
        foo
    });

    println!("{:?}", a)
}

I used it to narrow down where my problems come from, so I am even using the postcard Slice here:

error: lifetime may not live long enough
   --> src/main.rs:110:9
    |
102 |     let a: Yoke<TestStruct<_>, Vec<u8>> = Yoke::attach_to_cart(v, |x| {
    |                                                                    - has type `&'1 [u8]`
...
110 |         foo
    |         ^^^ returning this value requires that `'1` must outlive `'static`

Here again the compiler thinks that the lifetime of B in the TestStruct is 'static and I cannot figure out why. If I replace the struct with something like this (and remove the generic inferred parameter for TestStruct in the type annotation when calling attach_to_cart), it works:

struct TestStruct<'a> {
    a: &'a [u8],
}

However, I would like the be generic over B: ByteSlice, because I can have an &[u8] or a &mut [u8] in case I want to serialize this struct later. (I think...)

In my opinion B should just get inferred to &'1 [u8] in the closure and then Yoke does its magic to erase it and turn it into a 'static &[u8]. So why does the compiler think, that B lives so long? I tried building a small reproducer in a Rust playground, but I couldn't figure out how and I don't understand where that long lifetime is coming from. Any ideas?

edit: Wait, I think I know where this comes from: It's the Yokeable derive macro. I think this is what adds a B: 'static bound to its implementation.

edit2: Yes, that's exactly where it comes from. Now how do I get rid of it? I have the strong feeling right now that I can't do that. :D

[sffc]

First make sure it works without traits involved. Then, after you've confirmed that your structure works with concrete types, when you start adding traits, you may need to use prove_covariance_manually as documented below

https://docs.rs/yoke/latest/yoke/derive.Yokeable.html