This project describes how to build a soft-core RiscV based system with multiple Ethernet interfaces on a Digilent Arty-S7-50 with a Xilinx Spartan 7 FPGA using Python based Migen HDL and LiteX SoC builder.
Prototype of the project
More evolved version with up to four Ethernet interfaces and a PCB. The KiCad project files are included in this repository.
Download schematic as PDF.
It uses multiple Texas Instruments DP83848 based RMII Ethernet PHYs from Waveshare. In addition a SD-Card PMOD completes the system.
Note: in the PCB version the Seven Segment Display was removed due to I/O limitations.
More infos about this project can be found in the accompanying blog posts here and here.
To build the FPGA bitstream, the SoC BIOS and the sample bare metal demo application you need some prerequisites installed:
- Xilinx Vivado
- Python >= 3.6
- LiteX
- Tested with git version (bddb170)
- Liteeth
- Tested with git version (dea3590)
- RiscV compiler toolchain
- OpenOCD
- xc3sprog (optional)
Note: As of LiteX version (bddb170) and Liteeth version (dea3590) no patches are needed for multiple ethernet support. If you use an older version you can find patches in the history.
The included make.py script is responsible for building and loading/flashing.
In addition a Makefile is used to handle dependencies and file changes.
Assuming that all prerequisites are correctly installed using make:
make Builds FPGA bitstream and BIOS software
make build-sw Builds BIOS
make build-fpga Builds FPGA bitstream
make conv Convert Python FHDL to Verilog
make load Loads bitstream into FPGA via serial port
make load-sw Loads firmware binary using serial port
make reload Same as "load" but does not rebuild in case of changes
make flash Stores bitstream into FPGA boot flash
make flash-sw Flash BIOS software
make reflash Same as "flash" but does not rebuild in case of changes
make term Opens LiteX serial lxterm console
make clean Cleans/removes all build artefacts
The make.py script is called from make (Makefile) and provides the following:
make.py --help shows basic help
make.py build converts LiteX/Migen FHDL into verilog,
starts Xilinx Vivado and generates FPGA bitstream
make.py build-sw same as 'build' but only builds BIOS
make.py build-fpga same as 'build' but only builds FPGA bitstream
make.py load loads the bitstream into FPGA
make.py flash flashes the bitstream into config flash
make.py flash-sw flashes the BIOS into config flash
make.py conv creates all artefacts (verilog, c sources etc.)
but does not compile
In addition to the FPGA bitstream (which also includes the BIOS) a sample
application is included in the firmware directory. It can be compiled with
the included Makefile. The resulting firmware.bin can be loaded from SD-Card
or via TFTP from a server.
To load from SD-Card create a FAT16 partition on it and place firmware.bin
renamed to boot.bin in the root directory.
To load from TFTP server connect Ethernet interface 1 to a switch or directly
to a host via a cros-over network cable. Interface 1 is per default configured
with IP4 address 10.0.0.50 and tries to load from TFTP server with address
10.0.0.100. These addresses are static and can only be changed by modifying
top.py and re-building.
Place firmware.bin renamed to BOOT.BIN in the root directory of the TFTP
server.
To preserve the limited I/Os of the Arty the SD-Card can be removed and the
firmware booted from the SPI flash. The Arty has 16MB SPI flash normally used
to configure the FPGA with a bitstream at boot.
As lang as bitstream size + firmware size < 15MB they can reside together in
the flash. Current configuration halfs the flash. First half used for the
bitsream second half for the firmware. The make flash-sw writes the
firmware.fbi to flash adress 0x800000.
Note: Due to BIOS ROM size limitations the option to boot from SPI flash has been disabled. Booting the system is possible via SPI SD-Card, TFTP and serial.
Build and flash the FPGA bitstream:
make
make flash
To boot from SD-Card place firmware.bin renamed to BOOT.BIN on SD-Card and
reset the board. To boot from SPI flash, flash with make flash-sw.
On an attached serial terminal (open one with make term) you should see the
following output:
[LXTERM] Starting...
__ _ __ _ __
/ / (_) /____ | |/_/
/ /__/ / __/ -_)> <
/____/_/\__/\__/_/|_|
Build your hardware, easily!
(c) Copyright 2012-2020 Enjoy-Digital
(c) Copyright 2007-2015 M-Labs
BIOS built on May 1 2020 22:04:43
BIOS CRC passed (f662f7bd)
Migen git sha1: --------
LiteX git sha1: 56aa7897
--=============== SoC ==================--
CPU: VexRiscv @ 100MHz
ROM: 32KB
SRAM: 4KB
L2: 8KB
MAIN-RAM: 262144KB
--========== Initialization ============--
Ethernet init...
Initializing SDRAM...
SDRAM now under software control
Read leveling:
m0, b0: |00000000000000000000000000000000| delays: -
m0, b1: |00000000000000000000000000000000| delays: -
m0, b2: |00000000000000000000000000000000| delays: -
m0, b3: |00000000000000000000000000000000| delays: -
m0, b4: |00000000000000000000000000000000| delays: -
m0, b5: |00000000000000000000000000000000| delays: -
m0, b6: |00001111111111111000000000000000| delays: 10+-06
m0, b7: |00000000000000000000000000000000| delays: -
best: m0, b6 delays: 10+-06
m1, b0: |00000000000000000000000000000000| delays: -
m1, b1: |00000000000000000000000000000000| delays: -
m1, b2: |00000000000000000000000000000000| delays: -
m1, b3: |00000000000000000000000000000000| delays: -
m1, b4: |00000000000000000000000000000000| delays: -
m1, b5: |10000000000000000000000000000000| delays: 00+-00
m1, b6: |00001111111111111000000000000000| delays: 10+-06
m1, b7: |00000000000000000000000000000000| delays: -
best: m1, b6 delays: 10+-06
SDRAM now under hardware control
Memtest OK
Memspeed Writes: 262Mbps Reads: 327Mbps
--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
SD Card via SPI Initialising
Reading MBR
Partition 1 Information: Active=0x00, Type=0x06, LBAStart=0x00000800
Partition 1 is FAT16
Read FAT16 Boot Sector
Jump Code: 0xeb 0x3c 0x90
OEM Code: [mkfs.fat]
Sector Size: 512
Sectors Per Cluster: 128
Reserved Sectors: 128
Number of Fats: 2
Root Dir Entries: 2048
Total Sectors Short: 0
Media Descriptor: 0xf8
Fat Size Sectors: 128
Sectors Per Track: 32
Number of Heads: 64
Hidden Sectors: 2048
Total Sectors Long: 4194304
Drive Number: 0x80
Current Head: 0x01
Boot Signature: 0x29
Volume ID: 0x3fab9b48
Volume Label: [LITEX ]
Volume Label: [FAT16 ]
Boot Sector Signature: 0xaa55
Reclocking from 400KHz to 14285KHz
sdCardFatTable = 0x4fff0000 Reading Fat16 Table (128 Sectors Long)
sdCardFat16RootDir = 0x4ffe0000 Reading Root Directory (128 Sectors Long)
Root Directory
File 1 [BOOT .BIN] @ Cluster 3 for 9532 bytes
Reading File [BOOT.BIN] into 0x40000000 : File starts at Cluster 3 length 9532
Clusters: 3
Executing booted program at 0x40000000
--============= Liftoff! ===============--
Hello firmware booting...
HELLO>
A Litescope debugger is available via eth2 with configured IP address of 10.0.0.42. The debug directory contains some scripts and samples on how to use it.
After board reset the embedded BIOS is started which performs some basic
initalization and then tries to load the firmware via available peripherals.
In this case first via serial port, then via SD-Card and last via Ethernet.
It found the BOOT.BIN hello firmware on the SD-Card and loads it.
HELLO >help
Available commands:
hello value - Returns value xor'ed with 0xff in FPGA
time - show current clock ticks
uptime - show system uptime in seconds
random [runs] - show random number generator values, default [500] runs
malloc [runs] - test UMM malloc, default [500] runs
ping - ping host (10.0.0.100) via network interface (eth0)
reboot - reboot CPU
help - this command
The hello firmware just shows how to develop bare metal applications using
console in- and output and accessing FPGA logic blocks. Details about the firmware can be found here.
Although there exists a OSCIN pin on the DP83848 PHY module which suggests a clock input, it is the output of the 50MHz oscillator available on the module. This oscillator directly clocks the DP83848 PHY and is available on the module connector as reference clock output.
The default LiteX Liteeth RMII MAC operates with the clock of a 50MHz PLL it instantiates assuming that it must provide the clock to the PHY. As the PHY modules use their own oscillator the corresponding clock pads are set to Null in the code so that the external clock will be used.
To synchronize the MAC with the PHY the PHYs clock can be used as input to the MAC instead of the default 50MHz PLL clock output.
On Xilinx Spartan7 FPGAs only the positive (P) version of clock capable (CC) pins can be used as they are internally special wired and there is only a limited number per I/O bank available. As each module has its own oscillator they should be synced to its corresponding MAC. This is somehow inefficient as normally multiple PHYs and MACs share a common clock. But to prevent modifying each module, each modules OSCIN is connected to one of four CC-P pin on the FPGA syncing its corresponding MAC.
Although a simple modification on the module makes it possible to share clocks. Removing the 0 Ohm resistor R1 makes it possible to clock the modules from outside via the OSCIN pin.
More infos about this project can be found in the accompanying blog posts here and here.