T₄P₄S, a multitarget P4₁₆ compiler

This is an experimental compiler for P4₁₆ and P4₁₄ files. For publications and more, see our homepage.

An older version of the compiler is also available.

Find out more about the P₄ language.

Getting started

Preparation

To start working with the compiler, simply download the bootstrap-t4p4s.sh script and execute it in the following way. The script installs all necessary libraries (DPDK, P4C, P4Runtime and more) and T₄P₄S itself, and sets up environment variables.

wget https://raw.githubusercontent.com/P4ELTE/t4p4s/master/bootstrap-t4p4s.sh
chmod +x bootstrap-t4p4s.sh
. ./bootstrap-t4p4s.sh

Notes.

• ⚠ The purpose of the script is to setup a convenient environment for you if you're just starting out. Therefore, you only need to execute it once. If you already have a working environment, you don't need to run the script again, and you probably shouldn't.

  • To get T₄P₄S only without the third party libraries: git clone --recursive https://github.com/P4ELTE/t4p4s
  • To update a previous T₄P₄S checkout, execute this command in its directory: git pull --recurse-submodules
  • If you rerun bootstrap-t4p4s.sh, the previously installed content will be put in a backup directory.

• Without the . at the beginning of the line, the environment variables will not be usable immediately.

  • In that case, you can either open a new terminal, or run . ./t4p4s_environment_variables.sh

• The script is intended to work on recent Debian based systems, e.g. the latest LTS edition of Linux Mint or Ubuntu.

  • Legacy systems such as Ubuntu 18.04 and 16.04 are not supported by the script, as they do not come with out-of-the-box support for sufficiently recent libraries (such as Meson 0.47.1 or newer required for building DPDK, or Python 3.8 or newer required for building T₄P₄S).
  • Even on legacy systems, the script may still be useful to you. You may disable stages of the script, and manually install the software.

  INSTALL_STAGE2_DPDK=no INSTALL_STAGE3_PROTOBUF=no INSTALL_STAGE4_P4C=no . ./bootstrap-t4p4s.sh

  • It is also useful if you are not interested in using P4Runtime features.

  INSTALL_STAGE5_GRPC=no . ./bootstrap-t4p4s.sh

• Setting the option DPDK_APPS=no lets you install DPDK without its applications. The SLIM_INSTALL=yes option further quickens installation by not compiling many of the device drivers of DPDK.

  DPDK_APPS=no . ./bootstrap-t4p4s.sh SLIM_INSTALL=yes . ./bootstrap-t4p4s.sh

• You may customise the build process further. As the P₄ library is rather slow to compile to begin with, CFLAGS isn't passed to it, but you may use the specific P4C_CFLAGS variable. Here is a sample configuration.

  DPDK_APPS=no CFLAGS="-march=native -mtune=native -O2" MESONFLAGS="-Db_pch=true -Ddebug=false -Doptimization=2 -Dstrip=true" P4C_CFLAGS="" . ./bootstrap-t4p4s.sh

• To see all possible options (including available stages), run the script the following way.

  ./bootstrap-t4p4s.sh showenvs

• If you happen to have some of the dependencies locally checked out, you can speed up the installation process by letting the script clone them locally.

  LOCAL_REPO_CACHE=/my/cache/dir . ./bootstrap-t4p4s.sh

• At this stage of development, T₄P₄S will not compile and run all P₄ programs properly. In particular, header stacks are not supported currently.

Overriding defaults.

• To increase efficiency, the script runs jobs on all cores of the system in parallel. Should you experience any problems (for example, your system may run out of memory), you can override the number of jobs.

  MAX_MAKE_JOBS=4 . ./bootstrap-t4p4s.sh

• By default, the script runs downloads in parallel. You can force it to work sequentially.

  PARALLEL_INSTALL=no . ./bootstrap-t4p4s.sh

• The script installs sensibly recent versions of DPDK, P4C and PROTOBUF by default. To use the most recent versions, set INSTALL_TOOL_VSN to newest.

  INSTALL_TOOL_VSN=newest . ./bootstrap-t4p4s.sh

• You can also individually override the version/date settings.

  DPDK_VERSION=20.05 . ./bootstrap-t4p4s.sh DPDK_VERSION=20.05 DPDK_FILEVSN=20.05.0 . ./bootstrap-t4p4s.sh P4C_COMMIT_DATE=2020-11-01 . ./bootstrap-t4p4s.sh

• The script uses clang, clang++ and lld by default if they are installed unless overridden. It also uses ccache.

  T4P4S_CC=gcc T4P4S_CXX=g++ T4P4S_LD=bfd . ./bootstrap-t4p4s.sh

First Example

You can test the correct installation of the compiler by running the following simple example ./t4p4s.sh %l2fwd.

This should result in the following message T₄P₄S switch running ./examples/p4_14/l2fwd.p4_14 exited normally.

The example runs a single test case offline (so no network card is needed to test it).

Options

In the t4p4s.sh script, options control the process of compilation and execution. The options are collected in the following phases.

1. By default, the colours.cfg, lights.cfg, the command line, examples.cfg and opts_${ARCH}.cfg are processed.
   • colours.cfg describes the available colours for output highlighting.
   • lights.cfg describes which colours are used in the terminal and in the switch output for highlighting.
   • examples.cfg sets options for each example.
   • opts_${ARCH}.cfg sets architecture specific options.
   • Currently, the only valid value for ${ARCH} is dpdk.
2. When the command line of the script is processed, anything not identifiable as a P₄ program is considered an option.
   • A P₄ program is the name of an existing file whose extension begins with p4.
   • Here, the options are separated by spaces, therefore their values are not allowed to contain spaces themselves.
3. Option files come in two flavours.
   • Some files (e.g. lights.cfg) contain an option definition on a single line.
   • Some files (e.g. examples.cfg) contain an example identifier and then any number of options on a line.
     • An example identifier is examplename@testcasename, or if @testcasename is not given, @std is used by default.
     • As in the case of the command line, options may not contain any spaces.
   • In both cases, empty lines (containing whitespace only) and comments (a ; not preceded by a number, until the end of the line) are ignored.

The format of option definitions is the following.

• Option names contain letters, numbers, - (dash), _ (underscore) and . (dot).

• Define myopt with the value myval. myopt=myval

• Define myopt that takes the default value on. myopt

• In the cases described above, spaces can be allowed. myopt=foo bar

• From this point on, myopt is ignored: it is considered not to be defined. ^myopt

• Define myopt with the value val only if mycondopt is defined at this time. mycondopt->myopt=val

• Define myopt with the value val only if mycondopt is not defined at this time. ^mycondopt->myopt=val

• Define myopt with the value val only if mycondopt is defined at this time, and its value is condval. mycondopt=condval->myopt=val

• In all of the above, += ++= can take the place of =. Instead of setting the option, they append to the current value: += with a space separator, ++= with a newline.

  • myopt=foo and then myopt+=bar is equivalent to myopt=foo bar

• For convenience, there are some abbreviations.

  Option given	Equivalent to
  @myvariant	variant=myvariant
  :myexample	example=myexample
  ::myexample	example=myexample dbg
  %myexample=mytestcase	example=myexample variant=test testcase=mytestcase
  %myexample	example=myexample variant=test testcase=test
  %%myexample=mytestcase	example=myexample variant=test verbose dbg testcase=mytestcase

Execution

The script returns an exit code of 0 if the execution was successful, and a non-zero value otherwise.

The script creates build/<example-name>.

• Under it, the directories build, srcgen and Makefile contain compilation artifacts, including the created switch executable.
• Log output is stored in log.
  • controller.log is the log output from the most recent controller execution.
  • For each execution, two log files are created.
    • The one with the simple .txt extension is a regular textual log.
    • The one with the lit.txt extension contains ANSI colour codes. Invoking cat on it, or using an appropriate viewer like SublimeANSI will show coloured output.
    • The logs of the most recent script execution are also available as last.txt and last.lit.txt.

Examples

Note that for non-testing examples, you will have to setup your network card, and probably adjust your configuration options.

1. Specify an example
   • Run an example with the default configuration ./t4p4s.sh :l2fwd
   • The program finds the source file under examples automatically, but you can also specify it manually ./t4p4s.sh ./examples/p4_14/l2fwd.p4_14 model=v1model
2. Execution phases, option settings
   • Specify one or more steps to be taken ./t4p4s.sh :l2fwd p4 ./t4p4s.sh :l2fwd c ./t4p4s.sh :l2fwd run
   • If no option is given, all phases (p4 c run) are active ./t4p4s.sh :l2fwd
   • Options can be given in any order (phases will always run in p4 c run order) ./t4p4s.sh :l2fwd p4 c ./t4p4s.sh :l2fwd c p4
   • All options have one parameter, which defaults to "on" ./t4p4s.sh :l2fwd p4=on c=on run=on
   • Prefixing an option with ^ suppresses it
     • Run only P4-to-C and C-to-switch compilation ./t4p4s.sh :l2fwd ^run
   • Set the controller configuration (the controller program takes it as a parameter) ./t4p4s.sh :l2fwd ctrcfg=my_ctr_opts.txt
3. Output options: highlighting, verbosity
   • Get monochrome (black-and-white) output, useful for scripting ./t4p4s.sh :l2fwd bw
   • Monochrome terminal, colour switch execution ./t4p4s.sh :l2fwd bw=terminal
   • Colour terminal, monochrome switch execution ./t4p4s.sh :l2fwd bw=switch
   • Verbose output for the terminal ./t4p4s.sh :l2fwd verbose
   • Verbose output for the switch ./t4p4s.sh :l2fwd dbg
   • Even more verbose output for the switch ./t4p4s.sh :l2fwd dbg=1
   • In addition, statistics can be displayed at the end ./t4p4s.sh :l2fwd dbg stats
   • For per-packet statistics, use stats=1 ./t4p4s.sh :l2fwd dbg stats=1
   • Suppress EAL messages from the switch output ./t4p4s.sh :l2fwd noeal
   • No output at all (both terminal and switch) except for errors ./t4p4s.sh :l2fwd silent
   • If the switch fails, runs it again in the debugger (by default, gdb) ./t4p4s.sh :l2fwd autodbg
4. Testing
   • Run a single test case
     • It runs offline: no network card is needed ./t4p4s.sh %l2fwd=test, ./t4p4s.sh %l2fwd=payload
     • Data for the test case is in examples/test-l2fwd.c
   • Stop the switch immediately upon encountering invalid data ./t4p4s.sh %l2fwd=payload strict
5. Hugepages
   • examples.cfg sets the required number of hugepages for each example
   • Set it to another value, e.g. make T₄P₄S use 1024 MB of hugepages ./t4p4s.sh %l2fwd hugemb=1024
   • You may specify the amount of hugepages instead of the desired size in megabytes, which is dependent on the size of the hugepages on your system ./t4p4s.sh %l2fwd hugepages=1024
   • Instruct t4p4s.sh not to modify the current number of hugepages (may cause problems if it is less than required for the example) ./t4p4s.sh %l2fwd hugepages=keep
   • Instruct t4p4s.sh to adjust the number of hugepages exactly to the requested amount (by default, the hugepage count is never decreased) ./t4p4s.sh %l2fwd hugeopt=exact
6. Environment variables
   • Many options can be overridden using environment variables. EXAMPLES_CONFIG_FILE="my_config.cfg" ./t4p4s.sh my_p4 @test EXAMPLES_CONFIG_FILE="my_config.cfg" COLOUR_CONFIG_FILE="my_colors.txt" P4_SRC_DIR="../my_files" ARCH_OPTS_FILE="my_opts.cfg" ./t4p4s.sh %my_p4 dbg verbose
   • To see which environment variables are available for customisation and what their default values are, run the following command. ./t4p4s.sh showenvs
   • If showenvs is not the first argument, it prints the argument values after they have been fully computed/substituted. ./t4p4s.sh %l2fwd showenvs
7. Controller
   • Set the controller manually ./t4p4s.sh :l2fwd ctr=l2fwd
   • Let the output of the controller be shown in a separate window. For this to work, gnome-terminal is used, as the more general x-terminal-emulator does not seem to work properly. ./t4p4s.sh %my_p4 ctrterm
8. Compilation: logging, recompilation, source file hints, optimisation
   • If you add extern void log(string s); to your P₄ file, calls to log("My message") will produce a line in the debug output. ./t4p4s.sh %my_p4 x_log
   • T₄P₄S caches compilation results and takes only those compilation steps that are necessary. Changes in included files are not taken into consideration, however. You can force full recompilation in this case. ./t4p4s.sh %my_p4 recompile
   • Inquisitive users may want to investigate the generated C source code. To help with this, T₄P₄S can generate comments that hint about the origins of a generated expression or statement. ./t4p4s.sh %my_p4 hint=all ./t4p4s.sh %my_p4 hint=nopath ./t4p4s.sh %my_p4 hint=noext ./t4p4s.sh %my_p4 hint=nofile
   • You may change the optimisation level for meson. See more details here. MESON_BUILDTYPE=release ./t4p4s.sh %my_p4
   • When using clang, you may make use of thin-lto. ./t4p4s.sh %my_p4 lto
   • You may pass extra options to meson like this. ./t4p4s.sh %my_p4 mopt+=-Db_ndebug=if-release mopt+=-Doptimization=3 mopt+=-Ddebug=false
   • Reduce the output file size by about 85% using upx. This is a reasonable compromise between compression ratio and speed. ./t4p4s.sh %my_p4 upx
   • To further adjust upx, you can pass options to the command. Note that --brute and --ultra-brute will probably break the executable. UPX_OPTS=-1 ./t4p4s.sh %my_p4 ./t4p4s.sh %my_p4 upx=--best
9. Miscellaneous options
   • Specify the P₄ version manually (usually decided by other options or P₄ file extension) ./t4p4s.sh :l2fwd vsn=14
   • Specify the used architecture model manually ./t4p4s.sh :l2fwd-gen model=v1model, ./t4p4s.sh :l2fwd-gen model=psa
   • Pass a test option to the P₄ compiler. This defines a macro called T4P4S_TEST_1 that is available during P₄ preprocessing. ./t4p4s.sh %my_p4 p4testcase=1

Testing

As described above, you can run individual test cases. To see detailed output about compilation and execution, use the following options.

./t4p4s.sh %%l2fwd=payload

To run all available test cases, execute ./run_tests.sh. You can also give this script any number of additional options.

./run_tests.sh verbose dbg stats

As its name implies, run_tests.sh runs each test case in the offline (nicoff, meaning no NIC present) mode. You may set the PREFIX and POSTFIX environment variables to make the script start t4p4s.sh with a different setup for the test case. For example, the following command tests whether the test cases compile in the online (nicon) mode, but it doesn't execute them.

PREFIX=: POSTFIX="" ./run_tests.sh ^run

Once the test cases are run, the script prints a summary of successful and failed test cases, grouped by the types of failures. You may indicate which tests are to be skipped by listing them in a file. See the default skip file, tests_to_skip.txt, for further details.

SKIP_FILE="my_skip_file" ./run_tests.sh verbose dbg stats

The batch file processes all P4 files from the folder examples and its subfolders (including symlinked ones) by default. You can override it like this.

START_DIR=my_examples/testcases/ ./run_tests.sh verbose dbg stats

The script can generate a HTML and JSON output of the test into the build/all-run-logs folder in the following way:

HTML_REPORT="yes" ./run_tests.sh   

There is also a possibility to ask the run_test to run multiple times a test if it not fails. There are cases that is not deterministic, so with this option you can test a case multiple times to find the problem.

RUN_COUNT=3 ./run_tests.sh

Using Docker with T₄P₄S

Currently, this area of T₄P₄S is under rewrite. Stay tuned.

Working with the compiler

HLIR

For more details on how to work with HLIR attribures, see the readme of the hlir16 submodule.

Special markers

The compiler uses the .py files inside the hardware_indep directory to generate Python code (saved with the extension .desugared.py under build/util/desugared_compiler), then executes the code to produce .c files. Under src/utils, files with the extension .sugar.py are also primarily used as code generators. The files are written with some syntactical sugar, which is described in the following.

• The files under hardware_indep have access to the global variable hlir16, which is the root of the representation.
  • The compiler silently prepares a generated_code global variable that starts out with an empty text. Usually, you do not want to manipulate it directly.
  • The files may contain the following markers. PyExpr stands for a Python expression.
    • #[ (insert generated code here): the code will be textually added to generated_code
    • #[ ... $my_var ...: the textual value of the Python variable my_var is inserted here
    • #[ ... ${PyExpr} ...: the code is evaluated, then its result will be inserted as text
    • #= PyExpr: the expression is evaluated, its result is inserted textually
      • an alternative to this is to use #[ ${Python expression}
    • #{ and #}: the same as #[, except that code between the two will be indented one level
      • the compiler expects that all opened #{ markers will have a proper corresponding #} marker
    • $${PyExpr} highlights the evaluated text using the default colour (T4LIGHT_default)
      • $$[mycolourname]{PyExpr} uses T4LIGHT_mycolourname as the colour of highlighting; these colours are defined in lights.cfg and must be listed in ALL_COLOURS of t4p4s.sh
      • $$[mycolourname]{PyExpr}{text} is the same as above, but text (which is just plain text) also appears in the highlighted part
      • $$[mycolourname][text1]{PyExpr}{text} is the same as above, but text1 (which is just plain text) also appears in the highlighted part
    • The generated C code can also use highlighting: use T4LIT(some text) or T4LIT(my header instance's name,hdrinst)
• The following capabilities are most useful inside the .sugar.py files, but are used in hardware_indep as well.
  • Functions whose name begin with gen_ are considered helper functions in which the above markers are usable.
    • Technically, they will have a local generated_code variable that starts out empty, and they will return it at the end.
    • In general, such functions will contain a single conditional with multiple clauses, with each clause generating a bit of code.
    • Usually, it's a good idea to have a function with the same name (without the gen_ part) that calls the function.
  • To facilitate finding the corresponding generator file, the desugared (generated) files contain line hints about the original file.
    • For types and expressions, these can be made inline, e.g. uint8_t /* codegen@123*/ means that the text uint8_t was generated by executing code on or around line 123 in codegen.sugar.py (in the directory src/utils).
    • Most of the code generate statements, they contain hints at the end of the line such as ... // actions@123
    • You can control the sugar style using file_sugar_style and the class SugarStyle (in compiler.py), see the end of codegen.sugar.py for usage examples.

Crypto devices for cryptography operations

It is hardcoded in the current prototype to create an OpenSSL-based virtual crypto device in DPDK in order to support encryption and decryption extern functions. The PMD for this virtual device is not compiled in DPDK by default.

To enable the OpenSSL crypto PMD, edit dpdk-19.02/config/common_base by changing

CONFIG_RTE_LIBRTE_PMD_OPENSSL=n

to

CONFIG_RTE_LIBRTE_PMD_OPENSSL=y

and do a rebuild on DPDK.

You can also activate a separate crypto node to run the commands with parameter crypto_node=openssl. If you run a crypto node, you have to configure an extra core that only will do the external job.

If you want to test with a constant time external function, you can set crypto_node=fake and set for example the time with fake_crypto_time=5000 that sets the external function to run until 5000 clock ticks.

An example call of async mode:

./t4p4s.sh :l2fwd-gen cores=4 ports=3x2 async_mode=pd crypto_node=fake fake_crypto_time=3000