Skip to content

Commit

Permalink
Merge pull request #22 from cryspen/jonas/testbench-crate
Browse files Browse the repository at this point in the history 
Separate crate for tests and benchmarks
  • Loading branch information
@jschneider-bensch
jschneider-bensch authored Nov 6, 2024
2 parents 7bfb419 + 362ec64 commit 387f04f
Show file tree
Hide file tree
Showing 42 changed files with 1,346 additions and 369 deletions.
115 changes: 115 additions & 0 deletions benchmarks.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Benchmark Results

All numbers reported here refer to cycle counts.

## ML-KEM

### ML-KEM 512

| Device | KeyGen[Debug] | KeyGen[Release] | Encaps[Debug] | Encaps[Release] | Decaps[Debug] | Decaps[Release] |
|---------------------------------|---------------|-----------------|---------------|-----------------|---------------|-----------------|
| Raspberry Pi 3 | | | | | | |
| Raspberry Pi 4 | | | | | | |
| [ESP32-S3] [^1] [^2] | | | | | | |
| [STM32-L4R5xx] (our Nucleo-144) | 4445020 | 771501 | 5044052 | 839800 | 6134089 | 839800 |
| [ESP32-C6] [^3] | | | | | | |
| [nRF52840-DK] | 5968523 | 764636 | 6806307 | 839812 | 8316331 | 1011669 |
| [nRF5340-DK] | | | | | | |
| [nRF52-DK] | | | | | | |
| - nRF52832 | 5789064 | 762595 | 6579357 | 838838 | 7998480 | 1010106 |
| - nRF52810 | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] |


### ML-KEM 768

| Device | KeyGen[Debug] | KeyGen[Release] | Encaps[Debug] | Encaps[Release] | Decaps[Debug] | Decaps[Release] |
|---------------------------------|---------------|-----------------|---------------|-----------------|---------------|-----------------|
| Raspberry Pi 3 | | | | | | |
| Raspberry Pi 4 | | | | | | |
| [ESP32-S3] [^1] [^2] | | | | | | |
| [STM32-L4R5xx] (our Nucleo-144) | 7085443 | 1273471 | 7996675 | 1395000 | 9421249 | 1591459 |
| [ESP32-C6] [^3] | | | | | | |
| [nRF52840-DK] | 9315152 | 1257679 | 10418524 | 1381088 | 12319554 | 1607698 |
| [nRF5340-DK] | | | | | | |
| [nRF52-DK] | | | | | | |
| - nRF52832 | 9434994 | 1276466 | 10691113 | 1401091 | 12626256 | 1630202 |
| - nRF52810 | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] |


### ML-KEM 1024

| Device | KeyGen[Debug] | KeyGen[Release] | Encaps[Debug] | Encaps[Release] | Decaps[Debug] | Decaps[Release] |
|---------------------------------|---------------|-----------------|---------------|-----------------|---------------|-----------------|
| Raspberry Pi 3 | | | | | | |
| Raspberry Pi 4 | | | | | | |
| [ESP32-S3] [^1] [^2] | | | | | | |
| [STM32-L4R5xx] (our Nucleo-144) | 10981246 | 1991691 | 11988292 | 2129442 | 13756274 | 2371826 |
| [ESP32-C6] [^3] | | | | | | |
| [nRF52840-DK] | 14_910_284 | 1_959_916 | 16_318_665 | 2_103_371 | 18_830_628 | 2_391_463 |
| [nRF5340-DK] | | | | | | |
| [nRF52-DK] | | | | | | |
| - nRF52832 | 14886756 | 1951395 | 16021505 | 2094650 | 18385162 | 2381655 |
| - nRF52810 | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] |


## ML-DSA
### ML-DSA 44

| Device | KeyGen[Debug] | KeyGen[Release] | Sign[Debug] | Sign[Release] | Verify[Debug] | Verify[Release] |
|---------------------------------|---------------|-----------------|-------------|---------------|---------------|-----------------|
| Raspberry Pi 3 | | | | | | |
| Raspberry Pi 4 | | | | | | |
| [ESP32-S3] [^1] [^2] | | | | | | |
| [STM32-L4R5xx] (our Nucleo-144) | 19524681 | 3476142 | 28743483 | 4825636 | 20929483 | 3695919 |
| [ESP32-C6] [^3] | | | | | | |
| [nRF52840-DK] | 25565997 | 3565042 | 37809350 | 4987219 | 27441288 | 3788007 |
| [nRF5340-DK] | | | | | | |
| [nRF52-DK] | | | | | | |
| - nRF52832 | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] |
| - nRF52810 | | | | | | |

### ML-DSA 65

| Device | KeyGen[Debug] | KeyGen[Release] | Sign[Debug] | Sign[Release] | Verify[Debug] | Verify[Release] |
|---------------------------------|---------------|-----------------|-------------|---------------|---------------|-----------------|
| Raspberry Pi 3 | | | | | | |
| Raspberry Pi 4 | | | | | | |
| [ESP32-S3] [^1] [^2] | | | | | | |
| [STM32-L4R5xx] (our Nucleo-144) | 36271715 | 6382246 | 47606417 | 7981373 | 37005196 | 6501876 |
| [ESP32-C6] [^3] | | | | | | |
| [nRF52840-DK] | 47073665 | 6638977 | 62247859 | 8364565 | 48188918 | 6767475 |
| [nRF5340-DK] | | | | | | |
| [nRF52-DK] | | | | | | |
| - nRF52832 | ❌ [^5] | ❌ [^6] | ❌ [^5] | ❌ [^6] | ❌ [^5] | ❌ [^6] |
| - nRF52810 | | | | | | |


### ML-DSA 87

| Device | KeyGen[Debug] | KeyGen[Release] | Sign[Debug] | Sign[Release] | Verify[Debug] | Verify[Release] |
|---------------------------------|---------------|-----------------|-------------|---------------|---------------|-----------------|
| Raspberry Pi 3 | | | | | | |
| Raspberry Pi 4 | | | | | | |
| [ESP32-S3] [^1] [^2] | | | | | | |
| [STM32-L4R5xx] (our Nucleo-144) | 60053867 | 10633875 | 137335754 | 21909905 | 61676143 | 10794553 |
| [ESP32-C6] [^3] | | | | | | |
| [nRF52840-DK] | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] | ❌ [^5] | ❌ [^4] |
| [nRF5340-DK] | | | | | | |
| [nRF52-DK] | | | | | | |
| - nRF52832 | ❌ [^5] | ❌ [^6] | ❌ [^5] | ❌ [^6] | ❌ [^5] | ❌ [^6] |
| - nRF52810 | | | | | | |


[STM32-L4R5xx]: https://www.st.com/en/microcontrollers-microprocessors/stm32l4r5zi.html?rt=db&id=DB3171
[ESP32-C6]: https://www.espressif.com/en/products/socs/esp32-c6
[STM32-L476RG]: https://www.st.com/en/microcontrollers-microprocessors/stm32l476rg.html?rt=db&id=DB2196
[nRF52840-DK]: https://www.nordicsemi.com/Products/nRF52840
[nRF5340-DK]: https://www.nordicsemi.com/Products/nRF5340
[nRF52-DK]: https://www.nordicsemi.com/Products/Development-hardware/nRF52-DK
[esp32-s3]: https://www.espressif.com/en/products/socs/esp32-s3
[^1]: https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/hw-reference/esp32s3/user-guide-devkitc-1.html
[^2]: https://www.espressif.com/sites/default/files/documentation/esp32-s3-wroom-1_wroom-1u_datasheet_en.pdf
[^3]: https://www.espressif.com/sites/default/files/documentation/esp32-c6-wroom-1_wroom-1u_datasheet_en.pdf
[^4]: The benchmark crashses due to insufficient stack space.
[^5]: Core locks up.
[^6]: Benchmark binary does not fit available flash space.
2 changes: 2 additions & 0 deletions libcrux-nrf52810/.gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/target
Cargo.lock
76 changes: 76 additions & 0 deletions libcrux-nrf52810/Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
[package]
authors = ["Jonas Schneider-Bensch <jonas@cryspen.com>"]
name = "libcrux-nrf52810"
edition = "2021"
version = "0.1.0"

[lib]
harness = false

# needed for each integration test
[[test]]
name = "integration"
harness = false

[dependencies]
cortex-m = { version = "0.7", features = ["critical-section-single-core"] }
cortex-m-rt = "0.7"
defmt = "0.3"
defmt-rtt = "0.4"
panic-probe = { version = "0.3", features = ["print-defmt"] }
cortex-m-semihosting = "0.5.0"
libcrux-iot-testutil = { path = "../libcrux-iot-testutil" }
libcrux-testbench = { path = "../libcrux-testbench" }
embassy-nrf = { version = "0.1.0", features = [ "nrf52810", "defmt", ] }
embedded-alloc = "0.6.0"

[dev-dependencies]
defmt-test = "0.3"

[features]
default = ["mldsa87", "mlkem1024"] # We want to benchmark the most expensive variants by default
mldsa44 = ["libcrux-testbench/mldsa44"]
mldsa65 = ["libcrux-testbench/mldsa65"]
mldsa87 = ["libcrux-testbench/mldsa87"]
mlkem512 = ["libcrux-testbench/mlkem512"]
mlkem768 = ["libcrux-testbench/mlkem768"]
mlkem1024 = ["libcrux-testbench/mlkem1024"]

# cargo build/run
[profile.dev]
codegen-units = 1
debug = 2
debug-assertions = true # <-
incremental = false
opt-level = 'z' # <-
overflow-checks = true # <-

# cargo test
[profile.test]
codegen-units = 1
debug = 2
debug-assertions = true # <-
incremental = false
opt-level = 3 # <-
overflow-checks = true # <-

# cargo build/run --release
[profile.release]
codegen-units = 1
debug = 2
debug-assertions = false # <-
incremental = false
lto = 'fat'
opt-level = 3 # <-
overflow-checks = false # <-

# cargo test --release
[profile.bench]
codegen-units = 1
debug = 2
debug-assertions = false # <-
incremental = false
lto = 'fat'
opt-level = 3 # <-
overflow-checks = false # <-

32 changes: 32 additions & 0 deletions libcrux-nrf52810/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Libcrux ML-KEM / DSA on nRF52480

This setup is based on the app template found at [https://github.com/knurling-rs/app-template].

## Dependencies

#### 1. `flip-link`:

```console
$ cargo install flip-link
```

#### 2. `probe-rs`:

``` console
$ # make sure to install v0.2.0 or later
$ cargo install probe-rs --features cli
```


## Running Benchmarks

With the device attached, run
```console
$ cargo rb mlkem --release
```
for a crude benchmark of ML-KEM 1024 and

```console
$ cargo rb mldsa_65 --release
```
for a crude benchmark of ML-DSA 65.
23 changes: 23 additions & 0 deletions libcrux-nrf52810/memory.x
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/* Linker script for the nRF52 - WITHOUT SOFT DEVICE */
MEMORY
{
/* NOTE K = KiBi = 1024 bytes */
FLASH : ORIGIN = 0x00000000, LENGTH = 192K
RAM : ORIGIN = 0x20000000, LENGTH = 24K
}

/* This is where the call stack will be allocated. */
/* The stack is of the full descending type. */
/* You may want to use this variable to locate the call stack and static
variables in different memory regions. Below is shown the default value */
/* _stack_start = ORIGIN(RAM) + LENGTH(RAM); */

/* You can use this symbol to customize the location of the .text section */
/* If omitted the .text section will be placed right after the .vector_table
section */
/* This is required only on microcontrollers that store some configuration right
after the vector table */
/* _stext = ORIGIN(FLASH) + 0x400; */

/* Size of the heap (in bytes) */
/* _heap_size = 1024; */
42 changes: 42 additions & 0 deletions libcrux-nrf52810/src/bin/mldsa.rs
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#![no_main]
#![no_std]

use cortex_m::peripheral::Peripherals;
use libcrux_nrf52810 as board; // global logger + panicking-behavior + memory layout

extern crate alloc;

use core::ptr::addr_of_mut;
use embedded_alloc::LlffHeap as Heap;

#[global_allocator]
static HEAP: Heap = Heap::empty();

#[cortex_m_rt::entry]
fn main() -> ! {
use libcrux_iot_testutil::*;

// Initialize the allocator BEFORE you use it
{
use core::mem::MaybeUninit;
const HEAP_SIZE: usize = 1024;
static mut HEAP_MEM: [MaybeUninit<u8>; HEAP_SIZE] = [MaybeUninit::uninit(); HEAP_SIZE];
unsafe { HEAP.init(addr_of_mut!(HEAP_MEM) as usize, HEAP_SIZE) }

let mut peripherals = Peripherals::take().unwrap();
peripherals.DCB.enable_trace();
peripherals.DWT.enable_cycle_counter();
}

// set up the test config
let test_config = TestConfig {
core_freq: board::COREFREQ,
only_names: alloc::vec![],
early_abort: false,
benchmark_runs: 5,
};

libcrux_testbench::mldsa::run_benchmarks(test_config);

board::exit()
}
41 changes: 41 additions & 0 deletions libcrux-nrf52810/src/bin/mlkem.rs
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#![no_main]
#![no_std]

use cortex_m::peripheral::Peripherals;
use libcrux_nrf52810 as board; // global logger + panicking-behavior + memory layout

extern crate alloc;

use core::ptr::addr_of_mut;
use embedded_alloc::LlffHeap as Heap;

#[global_allocator]
static HEAP: Heap = Heap::empty();

#[cortex_m_rt::entry]
fn main() -> ! {
use libcrux_iot_testutil::*;
// Initialize the allocator BEFORE you use it
{
use core::mem::MaybeUninit;
const HEAP_SIZE: usize = 1024;
static mut HEAP_MEM: [MaybeUninit<u8>; HEAP_SIZE] = [MaybeUninit::uninit(); HEAP_SIZE];
unsafe { HEAP.init(addr_of_mut!(HEAP_MEM) as usize, HEAP_SIZE) }

let mut peripherals = Peripherals::take().unwrap();
peripherals.DCB.enable_trace();
peripherals.DWT.enable_cycle_counter();
}

// set up the test config
let test_config = TestConfig {
core_freq: board::COREFREQ,
only_names: alloc::vec![],
early_abort: false,
benchmark_runs: 5,
};

libcrux_testbench::mlkem::run_benchmarks(test_config);

board::exit()
}
53 changes: 53 additions & 0 deletions libcrux-nrf52810/src/lib.rs
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#![no_main]
#![no_std]

use cortex_m_semihosting::debug;

use defmt_rtt as _; // global logger

use embassy_nrf as _; // memory layout

use panic_probe as _;

pub const COREFREQ: u32 = 4_000_000;

// same panicking *behavior* as `panic-probe` but doesn't print a panic message
// this prevents the panic message being printed *twice* when `defmt::panic` is invoked
#[defmt::panic_handler]
fn panic() -> ! {
cortex_m::asm::udf()
}

/// Terminates the application and makes a semihosting-capable debug tool exit
/// with status code 0.
pub fn exit() -> ! {
loop {
debug::exit(debug::EXIT_SUCCESS);
}
}

/// Hardfault handler.
///
/// Terminates the application and makes a semihosting-capable debug tool exit
/// with an error. This seems better than the default, which is to spin in a
/// loop.
#[cortex_m_rt::exception]
unsafe fn HardFault(_frame: &cortex_m_rt::ExceptionFrame) -> ! {
loop {
debug::exit(debug::EXIT_FAILURE);
}
}

// defmt-test 0.3.0 has the limitation that this `#[tests]` attribute can only be used
// once within a crate. the module can be in any file but there can only be at most
// one `#[tests]` module in this library crate
#[cfg(test)]
#[defmt_test::tests]
mod unit_tests {
use defmt::assert;

#[test]
fn it_works() {
assert!(true)
}
}
Loading

0 comments on commit 387f04f

Please sign in to comment.