Chipyard is an open-source framework for designing, simulating, and testing RISC-V processors and systems-on-chip (SoCs). It comes with a few flavors of RISC-V cores, including Rocket, BOOM, and others. It also comes with a lot of pre-built components and tools to help you get started quickly. Because Chipyard supports building production-grade core configurations that can work with industry standard EDA tools, as well as hobbyist-grade verilator models and FPGA targets, it is a great choice for both learning and professional development. However, this also makes the learning curve a bit steep for beginners.

When I initially started working with Chipyard, I made the naive assumption that I could simply clone the repository, run a few commands, and have a working RISC-V processor in no time. Little did I know that I will have to go implement a fix deep within the RocketCore CSRs to get this running. That was a fun learning experience though, and I am documenting the entire journey here for anyone else who might be interested in replicating this setup.

TL;DR Incident report

Problem: OpenOCD could not halt the Rocket Core, showing “Hart failed to halt” error, completely blocking all JTAG debugging.

Root Cause: A timing race condition where the TLB checked debug status one cycle before it was updated, causing it to deny access to the Chipyard debug ROM at address 0x800.

Fix: Changed io.status.debug from being driven by a registered signal (reg_debug) to a combinational signal (trapToDebug), making debug status visible in the same cycle as the debug exception.

Impact: Enables JTAG debugging to work correctly. The fix is minimal (one line), has no performance cost, and aligns with the architectural intent that debug mode should be visible immediately when a debug exception is taken.

I will go into more details of how I fixed this issue later in the post.

Objectives and Background

The objective we are setting out to achieve is to:

1. Build a simple RISC-V processor using Chipyard.
2. Simulate it using verilator.
3. Write, and build a custom test program to run on our core.
4. Run the test program on the simulated core.
5. Attach a debugger to the running core.
6. Step through the program using the debugger.

Yeah, as I stated earlier, I was naive. But hey, where is the fun in taking the easy route ?

I thought I knew how RISC-V debug works. But I had to learn a lot more about how RISC-V debug works in order to get everything working with Chipyard.

When I first brought up a rocket core back in 2020 on an FPGA to boot linux, Chipyard was in its infancy and I used the rocket core repo directly. You can find my old blog post about that here. Since then, Chipyard has matured a lot and now provides a much better framework for building RISC-V processors and systems.

Chipyard is built on top of several other open-source projects. The documentation says:

Chipyard is a framework for designing and evaluating full-system hardware using agile teams. It is composed of a collection of tools and libraries designed to provide an integration between open-source and commercial tools for the development of systems-on-chip

It is essentially a one-stop-shop for RISC-V SoC design. It integrates with several other projects including Rocket Chip, BOOM, FireSim, and more.

Environment setup

To install dependencies with the following steps:

# Update System
sudo apt-get update
sudo apt-get upgrade -y

# Install Build Tools
sudo apt-get install -y build-essential bison flex software-properties-common curl git \
    libgmp-dev libmpfr-dev libmpc-dev zlib1g-dev vim device-tree-compiler \
    libboost-regex-dev libboost-system-dev \
    libtool libtool-bin autoconf automake pkg-config texinfo gperf libusb-1.0-0-dev

Chipyard uses Conda to manage its dependencies.

# Download Miniforge (conda alternative)
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"

# Install
bash Miniforge3-$(uname)-$(uname -m).sh -b

# Initialize conda
source ~/miniforge3/bin/activate
conda init

# IMPORTANT: Restart your terminal after this step

After restarting terminal:

# Install conda-lock (CRITICAL for Chipyard build-setup)
conda install -n base -c conda-forge conda-lock

Next, clone the Chipyard repository and initialize the submodules:

git clone [https://github.com/ucb-bar/chipyard.git](https://github.com/ucb-bar/chipyard.git)
cd chipyard
git checkout 1.13.0



# Initialize Submodules and build setup
./build-setup.sh riscv-tools

#build openocd
./scripts/build-openocd.sh

# Source the environment variables
source env.sh

This script will take a while to complete as it downloads dependencies and submodules and builds a few.

Understanding Chipyard structure and configurations

Before we proceed to build our RISC-V processor, it’s important to understand the structure of the Chipyard repository. Understanding the directory layout helped me navigate a few issues I ran into later on.

I am not doing an extensive walkthrough of the entire repo, but here are some of the directories I looked into to get a sense of how things are organized:

• generators/: Contains the scala code for generating RISC-V cores and SoCs.
  • rocket-chip/: The heavily parametrized rocket core generator code written in scala that is passed on to chisel to generate verilog.
    • /src/main/scala/rocket/: Contains the rocket core specific code. CSR.scala is a good example to start.
  • rocket-chip-blocks/: Contains peripheral blocks for the rocket core.
    • src/main/scala/devices/uart/: Contains the UART peripheral scala code.
  • chipyard/src/main/scala/config: Contains the configuration files for different SoC builds.
    • RocketConfigs.scala: Contains various rocket core configurations. More on this later below.
• toolchains/: Contains the RISC-V toolchain installation setup during build-setup.
• sims/verilator/: Contains verilator simulation environment setup and testbenches.

The RocketConfigs.scala file contains various configurations for the rocket core. Each configuration is a class that extends from a base configuration class and overrides certain parameters to customize the core. This is used by the build system to generate different core variants.

For instance the RocketConfig class is defined as follows:

RocketConfig
├── Rocket Tile (1x Large Core)
│   ├── 64-bit RISC-V ISA (RV64IMAFDC)
│   ├── L1 Instruction Cache (I$)
│   ├── L1 Data Cache (D$)
│   ├── Hardware FPU
│   ├── Hardware Multiply/Divide
│   └── MMU with Virtual Memory
│
├── Memory Hierarchy
│   ├── L2 Cache (SiFive Inclusive Cache)
│   ├── 64 KiB On-chip Scratchpad (@ 0x08000000)
│   ├── 1x AXI4 Memory Channel
│   └── Serial TileLink Interface
│
├── Bus Interconnect (Coherent, Hierarchical)
│   ├── SBUS - System Bus (500 MHz)
│   ├── MBUS - Memory Bus (500 MHz)
│   ├── PBUS - Peripheral Bus (500 MHz)
│   ├── FBUS - Front Bus (500 MHz)
│   └── CBUS - Control Bus (500 MHz)
│
├── Peripherals & I/O
│   ├── UART
│   ├── Boot ROM
│   ├── GPIO
│   ├── SPI Flash Interface
│   └── Boot Address Register
│
└── Debug Infrastructure
    ├── JTAG Debug Module
    ├── System Bus Access (SBA)
    └── Debug Memory (8 data words)

Since we will be using verilator to simulate our core, our starting point is the Makefile in the sims/verilator directory. Verilator is a great tool for simulating hardware designs as it converts Verilog code into C++ code, which can then be compiled and executed. This allows for fast simulation speeds and allows easy software control over the simulation. The preferred configuration for verilator simulations in Chipyard is the FastRTLSimRocketConfig, which is optimized for fast simulation speeds by removing the TileLink protocol checkers/monitors to reduce simulation overhead and improve simulation speed.

Note: I faced some additional issues when using the RocketConfig configuration with the TileLink monitors asserting when when OpenOCD performs byte-level memory writes. Using the FastRTLSimRocketConfig configuration resolved these issues without needing any additional code changes.

The next step is to build the verilator simulation using the desired configuration.

Chipyard design hierarchy and boot flow

Sim hirearchy and testbench structure

To build the verilator simulation, we issue a make command from within the sims/verilator directory. We can pass the CONFIG variable to the make to point the build to FastRTLSimRocketConfig configuration for our simulation.

Since Verilator simulates the hardware design, the makefile is setup to wrap the design in a testbench that provides the necessary stimulus and environment for the design to operate correctly. Understanding the design hierarchy is important to understand how the testbench interacts with the core.

The makefile first invokes the Chisel build system to generate the Verilog code for the specified configuration. The generated Verilog code is then compiled by Verilator along with the testbench code to create the final simulation binary.

The Chisel build command used in the makefile looks like this:

java -cp <classpath> chipyard.Generator \
  --target-dir <build_dir> \
  --name chipyard.harness.FastRTLSimRocketConfig \
  --top-module chipyard.harness.TestHarness \
  --legacy-configs chipyard:FastRTLSimRocketConfig

The overall module hirearchy of the generated design looks like this:

These are some pointers to the Code organisation resulting in this hireacrchy:

• generators/chipyard/src/main/scala/harness/TestHarness.scala: Defines the TestHarness module that instantiates ChipTop and connects the simulation models.
• generators/chipyard/src/main/scala/ChipTop.scala: Defines the ChipTop module that instantiates the DigitalTop system.

Boot and execution flow in the Chipyard Verilator sim

When we run a Verilator simulation, the core starts in reset, executes a bootrom, waits for code to be loaded via TSI (Test Serial Interface), and then jumps to execute that code.

generators/rocket-chip/src/main/resources/vsrc/TestDriver.v is the top-level Verilog testbench that instantiates the TestHarness module and provides clock/reset generation, memory model, TSI interface, JTAG interface, and UART console. This is how the hardware hireatcgy looks like:

TestDriver (Verilog wrapper)
└─ TestHarness (Chisel Module)
    ├─ ChipTop (LazyModule - the DUT)
    │   ├─ system (DigitalTop)
    │   │   ├─ bootrom (at address 0x10000)
    │   │   │   └─ Contains boot code in bootrom.img
    │   │   ├─ bootaddr_reg (at address 0x1000)
    │   │   │   └─ Holds boot address (default: 0x80000000)
    │   │   ├─ tile_prci_domain
    │   │   │   └─ tile (RocketTile with Rocket core)
    │   │   │       └─ Starts at reset vector (0x10000)
    │   │   └─ fbus (Front bus for serial TileLink)
    │   └─ Ports (serial_tl, uart, debug, etc.)
    │
    └─ Harness Components (simulation models)
        ├─ SerialRAM (harness-side memory)
        │   ├─ TSIToTileLink converter
        │   └─ TL RAM/ROM models
        └─ SimTSI (C++ DPI module)
            └─ Connects to SerialTL port

The Boot Address Register is set in generators/testchipip/src/main/scala/boot/BootAddrReg.scala and is memory-mapped at address 0x1000. It holds the address where the core will jump to after the bootrom execution. By default, it is set to 0x80000000, which is the start of DRAM.

generators/testchipip/src/main/resources/testchipip/bootrom/bootrom.S contains the assembly code for the bootrom. Core immediately goes into a wait-for-interrupt (WFI) loop. It’s waiting for an MSIP (Machine Software Interrupt) to wake it up.

_hang:  // Reset vector entry point
  la a0, _start          // Load address of _start
  csrw mtvec, a0         // Set trap vector to _start
  li a0, 8               // MSIP bit (Machine Software Interrupt Pending)
  csrw mie, a0           // Enable MSIP in mie CSR
  csrs mstatus, a0       // Enable interrupts in mstatus

wfi_loop:                // WAIT FOR INTERRUPT
  wfi                    // Core goes to sleep here
  j wfi_loop

When we execute the sim, we will be passing an elf file to parse and load into the memory through a commandline argument. The code in chipyard/generators/testchipip/src/main/resources/testchipip/csrc/testchip_tsi.cc implements the TSI protocol to load the elf file into the simulated memory.

The boot sequence looks like this:

The key steps are:

1. Loads your program into DRAM (at 0x80000000) via TSI write commands
2. Writes boot address to bootaddr_reg (0x1000) via +init_write or automatically
3. Triggers MSIP for hart 0 (writes to CLINT at 0x2000000) to wake up the core that is waiting in WFI loop in the bootrom.

_start:  // Interrupt handler (set as mtvec)
  li a1, 0x2000000       // CLINT base address
  csrr a0, mhartid       // Get hart ID
  bnez a0, boot_core     // Multi-hart handling

boot_core_hart0:
  sw zero, 0(a1)         // Clear the MSIP interrupt
  li a0, BOOTADDR_REG    // Load 0x1000
  ld a0, 0(a0)           // Read boot address from bootaddr_reg
  csrw mepc, a0          // Set return address to boot addr (0x80000000)
  csrr a0, mhartid       // hartid for bootloader (a0 arg)
  la a1, _dtb            // DTB address for bootloader (a1 arg)
  li a2, 0x80
  csrc mstatus, a2       // Clear MPIE
  mret                   // Return from interrupt -> jumps to 0x80000000!

The code executing from the 0x80000000 address is your custom test program that you compiled and passed to the simulator. The core will now execute your program. To indicate program completion, you can use the mtohost csr to write a value back to the simulation environment.

// In your test program:
write_csr(mtohost, 1);  // Exit code 1 = success
// SimTSI detects this and returns exit code to the simulator

We can alternatively use the DMI/JTAG interface for program load, debugging etc. That is what we will use for our debugging setup with OpenOCD and GDB. WithSimJTAGDebug extends HarnessBinder in generators/chipyard/src/main/scala/harness/HarnessBinders.scala enables this mode.

The 1.13.0 release of Chipyard (Current Latest) has a bug in the RocketCore CSR code that prevents OpenOCD from halting the core correctly. We will need to apply a small fix to the RocketCore CSR code before building the simulation. Let’s dive into that debug story next. It also gives an idea of how to debug such issues in the future.

Fixing the debugger halt issue with a patch

Tl;DR / dont bore me with details

Apply this fix before building the verilator simulation. If you don’t want to go through the detals, skip the rest of this section.

To apply the fix automatically:

cd /generators/rocket-chip

# Backup original file
cp src/main/scala/rocket/CSR.scala src/main/scala/rocket/CSR.scala.orig

# Apply the fix
sed -i '1004s/io.status.debug := reg_debug/io.status.debug := trapToDebug/' \
    src/main/scala/rocket/CSR.scala

# Add comment above (optional, for documentation)
sed -i '1005i\  // FIX: Set debug status combinationally when taking debug exception\n  // This allows TLB to recognize debug mode BEFORE instruction fetch to debug ROM\n  // Without this, TLB denies access to debug ROM (0x800) causing halt failure' \
    src/main/scala/rocket/CSR.scala

Verify the change:

grep -A 3 "io.status.debug :=" src/main/scala/rocket/CSR.scala

Expected output:

  io.status.debug := trapToDebug
  io.status.isa := reg_misa

The debug Journey

It might be better to skip this section if you are not interested in the nitty-gritty details of the debugging process. This section also jumps ahead and uses some commands that are explained later in the post. Maybe you can come back to this section after reading the rest of the post.

While I Was trying to set up GDB and OpenOCD to debug my RISC-V core, I ran into an issue that took me a while to figure out. It started manifesting as TileLink monitor assertion failures during memory write operations initiated by OpenOCD and when unblocked by moving to FastRTLSimRocketConfig started showing the cores not halting when requested by openocd/gdb.

This is how it manifested:

Open On-Chip Debugger 0.12.0+dev-03904-gc8b0535b6 (2025-12-19-23:42)
Licensed under GNU GPL v2
Info : JTAG tap: riscv.cpu tap/device found: 0x00000001 (mfg: 0x000 (<invalid>), part: 0x0000, ver: 0x0)
Info : [riscv.cpu] datacount=8 progbufsize=16
Error: [riscv.cpu] Unable to halt. dmcontrol=0x80000001, dmstatus=0x00030ca2
Error: [riscv.cpu] Fatal: Hart 0 failed to halt during examine

From my previous experience with RocketCore, I had some idea of where to look for this. Like with all other hardware simulation debug, dumping the waveforms during simulation to look at the io_status_debug and related signals in the CSR module helped me understand what was going on.

Capturing VCD Waveforms

To capture waveforms, I rebuilt the simulator with debug symbols and VCD support:

cd sims/verilator
make CONFIG=FastRTLSimRocketConfig debug

Then ran it with VCD dumping enabled:

./simulator-chipyard.harness-FastRTLSimRocketConfig-debug \
    main.elf \
    +jtag_rbb_enable=1 \
    +vcdfile=debug_halt.vcd \
    +max-cycles=300000

The +max-cycles flag is important because WFI would cause an infinite loop otherwise.

Analyzing the Waveforms

Opening the VCD in GTKWave, I focused on these signals:

• trapToDebug - When debug exception is taken
• reg_debug - The registered debug status
• io_status_debug - What the TLB sees
• io_ptw_status_debug - TLB’s specific view
• core_PC - Program counter

When PC jumped to 0x800, trapToDebug went HIGH immediately, but io_status_debug was still LOW because it was driven by reg_debug which updates one cycle later.

The TLB checks debug status in the same cycle as the PC change. Seeing io_status_debug = 0, it denied access to the debug ROM region at 0x800, causing an instruction access fault.

Here’s the timing issue visualized:

Building the verilator simulation

Once the patch is applied, we can proceed to build the verilator simulation using the FastRTLSimRocketConfig configuration using the following commands:

cd sims/verilator
make CONFIG=FastRTLSimRocketConfig -j$(nproc)

This generates the verilator simulation binary named simulator-chipyard.harness-FastRTLSimRocketConfig in thesims/verilator/ directory.

The Fix

Looking at generators/rocket-chip/src/main/scala/rocket/CSR.scala around line 1004:

// Original buggy code:
io.status.debug := reg_debug  // ← Uses registered signal!

// The fix:
io.status.debug := trapToDebug  // ← Use combinational signal!

The signal trapToDebug is already computed and goes HIGH immediately when the debug exception is taken. By routing it directly to io.status.debug, the TLB sees the correct debug status in the same cycle.

After the fix `io.status.debug` goes HIGH in the same cycle as `trapToDebug`, allowing TLB to grant access to debug ROM.

WFI Limitation

After applying the CSR fix and rebuilding, I started running ingo another issue where the OpenOCD still could not halt the core. But the verbose logging was showing slightly different behavior now.

The error was the same:

Error: [riscv.cpu] Unable to halt

In the simulation, when verbose logging is enabled, I saw the core was stuck at a specific PC:

C0: 54 [1] pc=[0000000000010034] inst=[10500073] DASM(10500073)

That instruction is WFI in the bootrom.

BootROM waiting for MSIP interrupt in WFI loop

Taking a closer look at the bootrom code, I realized the core was stuck in the WFI loop waiting for an MSIP interrupt to wake it up. This works well when using TSI to load programs, as TSI triggers the MSIP after loading the program. But with JTAG debug, there is no automatic MSIP trigger.

When we run with a normal program with

  ./simulator-chipyard.harness-FastRTLSimRocketConfig main.elf +jtag_rbb_enable=1

The sequence is:

1. Core boots, enters bootrom at 0x10000
2. Bootrom sets up trap handler
3. Bootrom enters WFI loop waiting for MSIP from HTIF/TSI
4. HTIF/TSI would normally: - Load your program to 0x80000000 - Write boot address register - Send MSIP interrupt to wake core
5. Core wakes from WFI, takes MSIP interrupt, handler jumps to 0x80000000

But with JTAG debug:

1. Core boots, enters bootrom
2. Bootrom enters WFI waiting for MSIP
3. User tries to attach debugger
4. Debug interrupt arrives
5. PROBLEM: HTIF never sent MSIP because you’re trying to use JTAG instead!
6. Bootrom is waiting for MSIP specifically
7. Even though debug interrupt might wake from WFI temporarily, the bootrom code just… stays in its WFI loop forever

The solution is to pass BINARY=none when runnning the simulation and manually load the program via GDB after attaching. This way, the test harness TSI immediately triggers the MSIP after reset, allowing the core to exit WFI and be ready for debugging.

Building the Verilator Simulation

Important: Before building, make sure you’ve applied the CSR fix described in the previous section. The fix must be in place before you build, as it’s compiled into the simulator binary.

Once the patch is applied, we can proceed to build the verilator simulation using the FastRTLSimRocketConfig configuration using the following commands:

cd sims/verilator
make CONFIG=FastRTLSimRocketConfig -j$(nproc)

Building Source Code Test Programs

To build test programs to run on our RISC-V core, we can use the RISC-V toolchain that was installed during the Chipyard setup. The toolchain includes riscv64-unknown-elf-gcc for compiling C code into RISC-V binaries.

mkdir -p ~/riscv-test
cd ~/riscv-test
# Create a simple test program

This is out test ptogram main.c:

#include <stdint.h>

// --- Magic Symbols for Chipyard/HTIF ---
// The harness looks for these to know the CPU is alive and to handle exit codes.
volatile uint64_t tohost __attribute__((section(".htif")));
volatile uint64_t fromhost __attribute__((section(".htif")));
// ---------------------------------------

volatile int counter = 0;

int main() {
    int a = 5;
    int b = 10;

    while(1) {
        counter++;
        a = a + b;
        if (counter > 100) {
            counter = 0;
        }
    }
    return 0;
}

To build it, we can use the following command:

cd ~/chipyard/chipyard
source env.sh  # Always source first!

cd ~/riscv-test

# Compile main program (with debug symbols)
riscv64-unknown-elf-gcc -O0 -g -static -specs=nano.specs \
    -Wl,-Ttext=0x80000000 -o main.elf main.c

We intentionally skipped using uart or printing a message to keep things simple. The program just increments a counter in an infinite loop taht we can observe in the debugger in the next steps.

Running the simulation with OpenOCD and GDB

To enable debugging with OpenOCD and GDB, we need to configure OpenOCD to connect to the verilator simulation’s JTAG interface. We will create a configuration file named jtag_sim.cfg with the following content adjested for our source code location:

# Adapter configuration
adapter speed 10000
adapter driver remote_bitbang
remote_bitbang host localhost
# !!! UPDATE THIS PORT FROM SIMULATOR OUTPUT !!!
remote_bitbang port 12345

# RISC-V target configuration
set _CHIPNAME riscv
jtag newtap $_CHIPNAME cpu -irlen 5

set _TARGETNAME $_CHIPNAME.cpu
target create $_TARGETNAME riscv -chain-position $_TARGETNAME

# With BINARY=none, CPU can halt properly during init
# No need to defer examination
# $_TARGETNAME configure -defer-examine

# --- Debug Configuration ---
# Force hardware breakpoints (no memory writes needed)
gdb breakpoint_override hard

# Use sysbus for memory access (faster, supports all memory regions)
# System bus access is enabled via WithDebugSBA in Chipyard config
riscv set_mem_access sysbus

# Disable virtual memory lookup (prevents random page table walks)
riscv set_enable_virt2phys off

# 5 second timeout for simulator latency
riscv set_command_timeout_sec 5

# Note: set_prefer_sba is not available in OpenOCD 0.12.0
# System bus will be used automatically via set_mem_access sysbus

init

# With BINARY=none, halt should work during init
halt

Note: The riscv set_mem_access sysbus setting is critical. Without it, GDB’s load command will fail with “Failed to write memory” errors. Chipyard’s AbstractConfig includes WithDebugSBA which enables system bus access in hardware.

We will also create a gdb_init.gdb file to automate GDB commands:

# GDB initialization script for Chipyard debugging
# Usage: riscv64-unknown-elf-gdb ~/riscv-test/main.elf -x ~/riscv-test/gdb_init.gdb

echo \n=== Chipyard RISC-V GDB Setup ===\n\n

# Increase timeout for slow simulator
set remotetimeout 300

# Connect using extended-remote (recommended by OpenOCD)
target extended-remote localhost:3333

echo Resetting and halting CPU...\n
monitor reset halt

echo Loading program...\n
load

echo \nProgram loaded. Entry point at 0x80000026\n

# Skip C runtime startup  and jump directly to main
echo Skipping C runtime startup, jumping directly to main...\n

# Set PC to main
set $pc = 0x800000b6

# Set up stack
set $sp = 0x80010000
set $fp = 0x80010000

# Set arguments to 0
set $a0 = 0
set $a1 = 0

echo \nNow at main():\n
info registers pc
x/5i $pc

echo \nSetting breakpoint in main loop...\n
hbreak *0x800000c2

echo \n=== Ready to debug! ===\n
echo Type 'continue' to run to breakpoint\n
echo Type 'stepi' to step one instruction\n
echo Type 'next' to step one source line\n
echo Type 'print counter' to examine variables\n\n

We can now run and attach the debugger. This requires a three terminal setup:

Three terminal setup: 1) Verilator sim, 2) OpenOCD, 3) GDB

1. Terminal 1: Run the Verilator simulation

    cd chipyard
    source env.sh
   
    cd sims/verilator
    ./simulator-chipyard.harness-FastRTLSimRocketConfig \
      none \
      +verbose \
      +jtag_rbb_enable=1
   

   Expected output:

   
    [UART] UART0 is here (stdin/stdout).
   
    This emulator compiled with JTAG Remote Bitbang client. To enable, use +jtag_rbb_enable=1.
    Listening on port 37823
    Attempting to accept client socket
   
   

2. Terminal 2: Run OpenOCD to connect to the simulation

   
    cd chipyard
    source env.sh
   
    cd ~/riscv-test
   
    # IMPORTANT: Update the port in jtag_sim.cfg first
    # Edit the file and change remote_bitbang_port to match the simulator's port
   
    vi jtag_sim.cfg  # or use your preferred editor
   
    # Start OpenOCD
   
    openocd -f jtag_sim.cfg
   

   Expected output:

    Open On-Chip Debugger 0.12.0+dev-04007-g3bed4c801 (2025-12-20-15:48)
    Licensed under GNU GPL v2
    For bug reports, read
            <http://openocd.org/doc/doxygen/bugs.html>
    Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'.
    force hard breakpoints
    Info : Initializing remote_bitbang driver
    Info : Connecting to localhost:34321
    Info : remote_bitbang driver initialized
    Info : Note: The adapter "remote_bitbang" doesn't support configurable speed
    Info : JTAG tap: riscv.cpu tap/device found: 0x00000001 (mfg: 0x000 (<invalid>), part: 0x0000, ver: 0x0)
    Info : [riscv.cpu] datacount=8 progbufsize=16
    Info : [riscv.cpu] Disabling abstract command reads from CSRs.
    Info : [riscv.cpu] Disabling abstract command writes to CSRs.
    Info : [riscv.cpu] Examined RISC-V core
    Info : [riscv.cpu]  XLEN=64, misa=0x800000000094112d
    [riscv.cpu] Target successfully examined.
    Info : [riscv.cpu] Examination succeed
    Info : [riscv.cpu] starting gdb server on 3333
    Info : Listening on port 3333 for gdb connections
    riscv.cpu halted due to debug-request.
    Info : Listening on port 6666 for tcl connections
    Info : Listening on port 4444 for telnet connections
   

3. Terminal 3: Run GDB to connect to OpenOCD and debug the core

      cd chipyard
      source env.sh
   
      cd ~/riscv-test
   
      riscv64-unknown-elf-gdb main.elf -x gdb_init.gdb
   

   Expected output:

      GNU gdb (GDB) 14.1
   
      Copyright (C) 2023 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      Type "show copying" and "show warranty" for details.
      This GDB was configured as "--host=x86_64-pc-linux-gnu --target=riscv64-unknown-elf".
      Type "show configuration" for configuration details.
      For bug reporting instructions, please see:
      <https://www.gnu.org/software/gdb/bugs/>.
      Find the GDB manual and other documentation resources online at:
          <http://www.gnu.org/software/gdb/documentation/>.
   
      For help, type "help".
      Type "apropos word" to search for commands related to "word"...
      Reading symbols from main.elf...
   
      === Chipyard RISC-V GDB Setup ===
   

There are hundreds if not thousands of tutorials on how to set up a load-balanced infrastructure in AWS. This post is just my notes from when I set up a load-balanced infrastructure in AWS. Needless to say that the details of the real production app has been changed for confidentiality. So, you might see a silly application being load balanced. But the concepts are the same.

Note that this is not one of those ultra high scale applications. But it is a good starting point to build a scalable and production-ready infrastructure in AWS that could serve a few million users.

NOTE: The Cloud Formation Template and Code for this article is available in GitHub at embeddedinn/aws_production_lb

Requirements

• Application development has been in Docker containers in GitHub and we will be using the same for deployment.
  • The app exposes services on two ports. One for the frontend and one for the backend.
• The application should be able to scale horizontally.
• Multiple availability zones should be supported. But to save cost, they will be brought up only when needed.
• VPCs and securoty groups should be setup to isolate the internal resources and expose them only to the load balancer.
  • The load balancer is configured as a VPC link on the API Gateway that exposes the services to the public.
• A load balancer and API Gateway should be used to distribute the traffic.

The complete setup as seen in AWS application composer would look like this:

AWS Application Composer view of the setup

Architecture

The Diagram below shows an approximation of the architecture that we will be setting up. Details will be discussed in the following sections.

Network Architecture Overview and Setup

This section provides a detailed explanation of the network architecture designed to securely and efficiently handle web traffic for an application deployed in AWS. The architecture leverages a VPC with public and private subnets, a NAT Gateway, and an internal NLB, with traffic routed through an API Gateway. This setup ensures that only the API Gateway is exposed to the internet, while all other resources remain securely isolated within the private subnet of the VPC.

Virtual Private Cloud (VPC)

The foundation of this architecture is a Virtual Private Cloud (VPC), that is logically isolated to deploy resources. The VPC is configured with a CIDR block of 10.0.0.0/16, providing a large address space (65,536 IP addresses) that can be further subdivided into smaller subnets. The VPC allows us to segment and secure the infrastructure by placing different components in either public or private subnets.

Subnets

• A public subnet (10.0.1.0/24) is a portion of the VPC’s address range that is exposed to the internet. Instances in this subnet can communicate directly with the outside world. The public subnet is also where the NAT Gateway and API Gateway are typically placed.
• A private subnet (10.0.2.0/24) is another segment within the VPC, but unlike the public subnet, it is isolated from direct internet access. Resources in the private subnet, such as the ECS tasks, EC2 instances, and the internal Load Balancer (NLB), do not have direct access to or from the internet. This enhances security by keeping your backend infrastructure hidden from the public internet.

Internet Gateway (IGW)

The Internet Gateway allows instances in the public subnet to communicate with the internet. It is attached to the VPC and enables outbound traffic from the public subnet to reach the internet. However, in this architecture, the Internet Gateway is used primarily for the NAT Gateway and the API Gateway, rather than directly exposing application resources.

NAT Gateway

The NAT Gateway is deployed in the public subnet to provide outbound internet access for resources in the private subnet. Instances in the private subnet cannot be accessed directly from the internet, but they can initiate outbound connections (e.g., to download software updates or access external APIs) through the NAT Gateway. This ensures that backend resources remain secure while still having access to necessary external services.

Route Tables

• The Public Route Table is associated with the public subnet. It contains a route that directs all outbound traffic (0.0.0.0/0) to the Internet Gateway, allowing instances in the public subnet (like the NAT Gateway and API Gateway) to communicate with the internet.

• The Private Route Table is associated with the private subnet. It directs outbound traffic to the NAT Gateway, enabling instances in the private subnet to access the internet indirectly, without exposing them to inbound internet traffic. This ensures that sensitive components like the ECS tasks and the Load Balancer remain isolated.

Internal Network Load Balancer (NLB)

The NLB is placed in the private subnet and configured as an internal load balancer (Scheme: “internal”). This means it is not exposed to the internet and can only be accessed from within the VPC or through other AWS services like the API Gateway. The internal NLB distributes incoming traffic across multiple backend resources (e.g., ECS tasks, EC2 instances) running in the private subnet. By keeping the NLB internal, the architecture adds a layer of security, ensuring that only authorized traffic (routed through the API Gateway) can reach the backend services.

API Gateway

The API Gateway is the sole internet-facing component in this architecture. It is deployed in the public subnet and acts as a reverse proxy, handling all incoming HTTP requests from the internet. The API Gateway is configured to forward these requests to the internal NLB, which then routes the traffic to the appropriate backend services. This setup ensures that the backend services are not directly exposed to the internet, enhancing security and providing a single entry point for all external traffic.

Security Groups

• The Load Balancer Security Group is configured to accept incoming traffic only from the API Gateway. This ensures that only requests routed through the API Gateway can reach the Load Balancer and, subsequently, the backend services.

• The API Gateway Security Group allows traffic from any IP address (i.e., the public internet) but restricts outgoing traffic to the internal NLB. This ensures that only the API Gateway can forward traffic to the Load Balancer, maintaining a secure and controlled flow of data.

• The EC2 Security Group allows incoming traffic from the Load Balancer Security Group, enabling communication between the Load Balancer and the EC2 instances. Outgoing traffic is allowed to any IP address, as the EC2 instances may need to access external resources.

• The ECS Security Group is similar to the EC2 Security Group but is specifically designed for ECS tasks. It allows incoming traffic from the Load Balancer Security Group and outgoing traffic to any IP address.

Traffic Flow

• Inbound Traffic : External requests from clients on the internet are directed to the API Gateway, which serves as the front door for the application. The API Gateway processes and proxies these requests, forwarding them to the internal NLB within the private subnet.
• Load Balancing : The NLB receives the traffic from the API Gateway and distributes it across multiple instances or containers running in the private subnet. This ensures that the application can handle varying loads by automatically scaling the number of backend resources.
• Outbound Traffic : Instances in the private subnet can initiate outbound connections to the internet via the NAT Gateway. This allows backend services to access external resources (such as downloading updates) without exposing themselves to inbound internet traffic.

The CloudFormation template below can be used to set up the network described here. Once created , you can see it in the console like this:

Generated Netwpork in AWS Console

AWSTemplateFormatVersion: "2010-09-09"
Description: "CloudFormation template to set up VPC, Subnets, NAT Gateway, and Internal Load Balancer."

Resources:
  # VPC
  VPC:
    Type: "AWS::EC2::VPC"
    Properties:
      CidrBlock: "10.0.0.0/16"
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: "Name"
          Value: "appVPC"

  # Internet Gateway
  InternetGateway:
    Type: "AWS::EC2::InternetGateway"
    Properties:
      Tags:
        - Key: "Name"
          Value: "InternetGateway"

  # Attach Internet Gateway to VPC
  AttachGateway:
    Type: "AWS::EC2::VPCGatewayAttachment"
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  # Public Subnet
  PublicSubnet:
    Type: "AWS::EC2::Subnet"
    Properties:
      VpcId: !Ref VPC
      CidrBlock: "10.0.1.0/24"
      AvailabilityZone: !Select [0, !GetAZs ""]
      MapPublicIpOnLaunch: true
      Tags:
        - Key: "Name"
          Value: "PublicSubnet"

  # Private Subnet
  PrivateSubnet:
    Type: "AWS::EC2::Subnet"
    Properties:
      VpcId: !Ref VPC
      CidrBlock: "10.0.2.0/24"
      AvailabilityZone: !Select [0, !GetAZs ""]
      Tags:
        - Key: "Name"
          Value: "PrivateSubnet"

  # Route Table for Public Subnet
  PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: "Name"
          Value: "PublicRouteTable"

  # Route Table for Public Subnet
  PublicRoute:
    Type: "AWS::EC2::Route"
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: !Ref InternetGateway

  # Associate Public Subnet with Route Table
  PublicSubnetRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTable

  # NAT Gateway
  NatEIP:
    Type: "AWS::EC2::EIP"
    Properties:
      Domain: "vpc"

  # NAT Gateway
  NatGateway:
    Type: "AWS::EC2::NatGateway"
    Properties:
      AllocationId: !GetAtt NatEIP.AllocationId
      SubnetId: !Ref PublicSubnet
      Tags:
        - Key: "Name"
          Value: "natGateway"

  # Route Table for Private Subnet
  PrivateRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: "Name"
          Value: "PrivateRouteTable"

  # Route Table for Private Subnet
  PrivateRoute:
    Type: "AWS::EC2::Route"
    Properties:
      RouteTableId: !Ref PrivateRouteTable
      DestinationCidrBlock: "0.0.0.0/0"
      NatGatewayId: !Ref NatGateway

  # Associate Private Subnet with Route Table
  PrivateSubnetRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      SubnetId: !Ref PrivateSubnet
      RouteTableId: !Ref PrivateRouteTable

  # Internal Load Balancer (Private)
  LoadBalancer:
    Type: "AWS::ElasticLoadBalancingV2::LoadBalancer"
    Properties:
      Name: "internalLoadBalancer"
      Subnets:
        - !Ref PrivateSubnet
      Scheme: "internal"
      LoadBalancerAttributes:
        - Key: "deletion_protection.enabled"
          Value: "false"
      SecurityGroups:
        - !Ref LoadBalancerSecurityGroup
      Type: "network"
      IpAddressType: "ipv4"
      EnforceSecurityGroupInboundRulesOnPrivateLinkTraffic: "off"
      Tags:
        - Key: "Name"
          Value: "internalLoadBalancer"

  # Security Group for Load Balancer
  LoadBalancerSecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties:
      GroupDescription: "Allow inbound traffic to Load Balancer"
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !GetAtt ApiGatewaySecurityGroup.GroupId
        - IpProtocol: tcp
          FromPort: 8000
          ToPort: 8000
          SourceSecurityGroupId: !GetAtt ApiGatewaySecurityGroup.GroupId
      SecurityGroupEgress:
        - IpProtocol: "-1"
          CidrIp: "0.0.0.0/0"
      Tags:
        - Key: "Name"
          Value: "LoadBalancerSecurityGroup"

  # Security Group for API Gateway
  ApiGatewaySecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties:
      GroupDescription: "Allow traffic from API Gateway to Load Balancer"
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: "0.0.0.0/0"
        - IpProtocol: tcp
          FromPort: 8000
          ToPort: 8000
          CidrIp: "0.0.0.0/0"
      SecurityGroupEgress:
        - IpProtocol: "-1"
          CidrIp: "0.0.0.0/0"
      Tags:
        - Key: "Name"
          Value: "ApiGatewaySecurityGroup"

  # Security Group for ECS Tasks
  ECSSecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties:
      GroupDescription: "Allow traffic from Load Balancer to ECS Tasks"
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !GetAtt LoadBalancerSecurityGroup.GroupId
        - IpProtocol: tcp
          FromPort: 8000
          ToPort: 8000
          SourceSecurityGroupId: !GetAtt LoadBalancerSecurityGroup.GroupId
      SecurityGroupEgress:
        - IpProtocol: "-1"
          CidrIp: "0.0.0.0/0"
      Tags:
        - Key: "Name"
          Value: "ECSSecurityGroup"

  # Security Group for EC2 Instances
  EC2SecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties:
      GroupDescription: "Allow traffic from Load Balancer to EC2 Instances"
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          SourceSecurityGroupId: !GetAtt LoadBalancerSecurityGroup.GroupId
        - IpProtocol: tcp
          FromPort: 8000
          ToPort: 8000
          SourceSecurityGroupId: !GetAtt LoadBalancerSecurityGroup.GroupId
      SecurityGroupEgress:
        - IpProtocol: "-1"
          CidrIp: "0.0.0.0/0"
      Tags:
        - Key: "Name"
          Value: "EC2SecurityGroup"

Application Deployment - Local Setup

Now that we have setup the network, we can deploy the application. But before that , we must be able to test it locally.

The application is a simple web app that consists of a HTML frontend and a python backend. The frontend is a simple HTML page that calls a backend API to show the API status. The application is containerized using Docker. The frontend will be served on port 80, and the backend will be served on port 8000.

This is a trivial application that we will use to demonstrate the setup of a multi-component production infrastructure. The actual application will be more complex and will have additional services and components.

Frontend

The index.html file contains the following content:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Frontend</title>
</head>

<body>
    <h1>Frontend</h1>
    <p id="data"></p>

    <script>
        // Set the API host to wherever the page is loaded from
        const api_host = `${window.location.origin}:8000/status`;

        fetch(api_host)
            .then(response => response.json())
            .then(data => {
                document.getElementById('data').innerText = "API STATUS: " + data.STATUS;
            })
            .catch(error => {
                console.error('Error:', error);
            });
    </script>
</body>
</html>

Note: We will later change the backend URL to the API gateway resource URL during deployment.

Backend

The backend exposes a simple FastAPI based API endpoint that is used by the frontend. The file structure of the backend service that we will setup in docker is as follows:

.
├── app
│   ├── __init__.py
│   └── main.py
└── static
    └── index.html
└── requirements.txt

__init.py is blank, and main.py contains the FastAPI code:

#main.py
from typing import Union
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
import threading
import uvicorn

backend = FastAPI()
frontend = FastAPI()

# CORS settings for the backend
backend.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],  # Allow all origins
    allow_credentials=True,
    allow_methods=["GET"],  # Allow only GET methods
    allow_headers=["*"],  # Allow all headers
)

@backend.get("/status")
def read_status():
    return {"STATUS": "UP"}

@backend.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}

frontend.mount("/", StaticFiles(directory="static", html=True), name="static")

def run_backend():
    uvicorn.run(backend, host="0.0.0.0", port=8000)

def run_frontend():
    uvicorn.run(frontend, host="0.0.0.0", port=80)

if __name__ == "__main__":
    backend_thread = threading.Thread(target=run_backend)
    frontend_thread = threading.Thread(target=run_frontend)

    backend_thread.start()
    frontend_thread.start()

    backend_thread.join()
    frontend_thread.join()

The requirements.txt file contains the following dependencies:

fastapi[standard]>=0.112.2
pydantic>=2.8.2
uvicorn>=0.30.6

The backend app can be run locally using the following commands:

python3 app/main.py

Docker Setup

Now that we have the frontend and backend components, we can create a docker image for them. We will wrap both the frontend and backend in a single Dockerfile. The Dockerfile is as follows:

FROM python:3.12-slim
WORKDIR /code
COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
COPY ./app /code/app
COPY ./index.html /code/static/index.html
CMD ["python3", "app/main.py"]
EXPOSE 80
EXPOSE 8000

the docker container can be built and run using the following commands:

docker build . -t app
docker run -p 80:80 -p 8000:8000 app:latest

AWS Deployment

Now that we have a working version of the application, we can deploy it to AWS. We will use AWS ECS to deploy the application in a scalable and fault-tolerant manner. The application will be deployed as a service in an ECS cluster. The docker image will be built and pushed to AWS ECR, and the ECS service will be configured to use this image.

Add the following statements to the cloudformation template to setup a private ECR repo to host the docker images.

  # ECR Repository
  ECRRepository:
    Type: "AWS::ECR::Repository"
    Properties:
      RepositoryName: "app_repository"
      ImageScanningConfiguration:
        ScanOnPush: true
      Tags:
        - Key: "Name"
          Value: "appRepository"

Update the stack with the following command:

aws cloudformation update-stack --stack-name App-Stack --template-body file://temlpate.yml  --profile embeddedinn

To push the image to the repository manually, navigate to ECR from the console and follow the steps under “View push commands”.

In later sections, we will setup a GitHub action to version control the application files and automate this process.

Capacity Provisioning

The next step is to provision the capacity for the ECS cluster. We will use an Auto Scaling Group to manage the EC2 instances that will run the ECS tasks. The Auto Scaling Group will be configured to launch instances in the private subnet of the VPC, ensuring that the backend services are isolated from the public internet.

The launch template will pull in Amazon Linux image optimized for ECS. This image runs the ECS agent and is pre-configured to work with ECS. The launch template will also specify the instance type, key pair, and security group for the EC2 instances. The template also defines the availability zone and subnet for the instances.

Using the launch template, we will define the Auto Scaling Group. We will set the max capacity to 10 instances, with a desired capacity of 0. This will ensure that no instances are running initially, and the Auto Scaling Group will scale up based on the demand from ECS tasks.

The autoscaling group will be then wrapped into a capacity provider. The capacity provider will be used by the ECS service to manage the capacity of the ECS cluster.

We will omit attaching a key pair to the instances as we will be using the ECS service to manage the instances.

The following CloudFormation template can be used to setup these items:

  # IAM Role for ECS Instance
  ECSInstanceRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "ecsInstanceRole"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "ec2.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
        - "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
        - "arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"

  # IAM Instance Profile for ECS Instance
  ECSInstanceProfile:
    Type: "AWS::IAM::InstanceProfile"
    Properties:
      InstanceProfileName: "ecsInstanceProfile"
      Roles:
        - !Ref ECSInstanceRole

  # Launch Template for ECS Instance
  ECSLaunchTemplate:
    Type: "AWS::EC2::LaunchTemplate"
    Properties:
      LaunchTemplateName: "ecsLaunchTemplate"
      LaunchTemplateData:
        ImageId: "ami-09f63ae8e9b15e9b2"
        InstanceType: "t2.micro"
        Monitoring:
          Enabled: true
        TagSpecifications:
          - ResourceType: "instance"
            Tags:
              - Key: "Name"
                Value: "appInstance"
        NetworkInterfaces:
          - AssociatePublicIpAddress: false
            DeleteOnTermination: true
            DeviceIndex: 0
            Groups:
              - !GetAtt EC2SecurityGroup.GroupId
        IamInstanceProfile:
          Arn: !GetAtt ECSInstanceProfile.Arn
        UserData: !Base64 |
          #!/bin/bash
          echo ECS_CLUSTER=appCluster >> /etc/ecs/ecs.config

  # ECS Auto Scaling Group
  ECSAutoScalingGroup:
    Type: "AWS::AutoScaling::AutoScalingGroup"
    Properties:
      AutoScalingGroupName: "ecsAutoScalingGroup"
      LaunchTemplate:
        LaunchTemplateId: !Ref ECSLaunchTemplate
        Version: !GetAtt ECSLaunchTemplate.LatestVersionNumber
      MinSize: 0
      MaxSize: 10
      HealthCheckType: "EC2"
      AvailabilityZones:
        - !Select [0, !GetAZs ""]
      TerminationPolicies:
        - Default
      HealthCheckGracePeriod: 0
      DesiredCapacity: 0
      VPCZoneIdentifier:
        - !Ref PrivateSubnet
      Tags:
        - Key: "Name"
          Value: "ecsAutoScalingGroup"
          PropagateAtLaunch: true
      ServiceLinkedRoleARN: !Sub "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"

  # ECS Capacity Provider
  ECSCapacityProvider:
    Type: "AWS::ECS::CapacityProvider"
    Properties:
      Name: "appECSCapacityProvider"
      AutoScalingGroupProvider:
        AutoScalingGroupArn: !Ref ECSAutoScalingGroup
        ManagedScaling:
          Status: "ENABLED"
          TargetCapacity: 100
          MinimumScalingStepSize: 1
          MaximumScalingStepSize: 10
          InstanceWarmupPeriod: 300
        ManagedTerminationProtection: "DISABLED"
        ManagedDraining: ENABLED

Note : Notice the userdata section in the launch template. This is used to configure the ECS agent on the instances. We are hardcoding the cluster name here. This is an item that could be parameterized.

Loadbalancer Target Group and listner Setup

A target group in the load balancer is used to route traffic to the backend services. The target group is associated with the ECS tasks that are running the backend services. The target group is configured to use the internal NLB as the load balancer, ensuring that traffic is routed to the backend services running in the private subnet. When ECS clusters scale up or down, the target group will automatically register or deregister the ECS tasks based on the demand.

We will setup two target groups, one for the frontend and one for the backend. The frontend target group will be associated with the frontend service running on port 80, and the backend target group will be associated with the backend service running on port 8000. These target groups will be also added to the load balancer as listeners.

  # Target Groups for Load Balancer
  FrontendTargetGroup:
    Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
    Properties:
      Name: "frontendTargetGroup"
      Port: 80
      Protocol: "TCP"
      VpcId: !Ref VPC
      TargetType: "instance"
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: "/"
      HealthCheckPort: "80"
      HealthCheckProtocol: "HTTP"
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2

  # Target Groups for Load Balancer
  BackendTargetGroup:
    Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
    Properties:
      Name: "backendTargetGroup"
      Port: 8000
      Protocol: "TCP"
      VpcId: !Ref VPC
      TargetType: "instance"
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: "/status"
      HealthCheckPort: "8000"
      HealthCheckProtocol: "HTTP"
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2

  # Listeners for Load Balancer
  FrontendListener:
    Type: "AWS::ElasticLoadBalancingV2::Listener"
    Properties:
      DefaultActions:
        - TargetGroupArn: !Ref FrontendTargetGroup
          Type: "forward"
      LoadBalancerArn: !Ref LoadBalancer
      Port: 80
      Protocol: "TCP"

  # Listeners for Load Balancer
  BackendListener:
    Type: "AWS::ElasticLoadBalancingV2::Listener"
    Properties:
      DefaultActions:
        - TargetGroupArn: !Ref BackendTargetGroup
          Type: "forward"
      LoadBalancerArn: !Ref LoadBalancer
      Port: 8000
      Protocol: "TCP"

ECS Cluster Setup

Now that we have a cluster capacity provider, we can define the tasks, services, and cluster for the ECS setup. A task definition defines the container image and port mappings for the ECS task and uses the default cluster capacity provider. The task definition will be used by the ECS service to run the containers in the ECS cluster.

The ECS service will be configured to use the capacity provider we defined earlier. This will ensure that the service can scale up and down based on the demand from the ECS tasks.

The ECS cluster will be created with the capacity provider and the service. The cluster will be associated with the VPC and the private subnet. The service tasks will register with the NLB listeners.

Note that the cluster name here should match the userdata in the launch template in order for the ECS agent to register the instances with the cluster.

  # Create a cluster to run the ECS tasks
  ECSCluster:
    Type: "AWS::ECS::Cluster"
    Properties:
      ClusterName: "appCluster"
      CapacityProviders:
        - !Ref ECSCapacityProvider
      DefaultCapacityProviderStrategy:
        - CapacityProvider: !Ref ECSCapacityProvider
          Weight: 1
          Base: 1

  #EC2 ECS task definition
  ECSTaskDefinition:
    Type: "AWS::ECS::TaskDefinition"
    Properties:
      RequiresCompatibilities:
        - "EC2"
      ContainerDefinitions:
        - Name: "appContainer"
          Image: !GetAtt ECRRepository.RepositoryUri
          PortMappings:
            - ContainerPort: 8000
              HostPort: 8000
              Protocol: "tcp"
            - ContainerPort: 80
              HostPort: 80
              Protocol: "tcp"
          Essential: true
          Memory: 512
      Family: "appTaskDefinition"

  # ECS Service
  ECSService:
    Type: "AWS::ECS::Service"
    Properties:
      Cluster: !Ref ECSCluster
      ServiceName: "appService"
      TaskDefinition: !Ref ECSTaskDefinition
      DesiredCount: 2
      DeploymentConfiguration:
        MaximumPercent: 100
        MinimumHealthyPercent: 50
      LoadBalancers:
        - ContainerName: "appContainer"
          ContainerPort: 8000
          TargetGroupArn: !Ref BackendTargetGroup
        - ContainerName: "appContainer"
          ContainerPort: 80
          TargetGroupArn: !Ref FrontendTargetGroup
      SchedulingStrategy: "REPLICA"
      DeploymentController:
        Type: "ECS"
      CapacityProviderStrategy:
        - CapacityProvider: !Ref ECSCapacityProvider
          Weight: 1
      Tags:
        - Key: "Name"
          Value: "appService"
        - Key: "LoadBalancer"
          Value: !Ref LoadBalancer

The tags includes a reference to the load balancer to ensure that the cloudfomration stack waits for the load balancer to be created before creating the service.

At the end of this process, we can see the load balancer in the console like this:

Load Balancer in AWS Console

Two tasks have been brought up in two different EC2 instances that we configured using the template. The target group performs health checks on the tasks and routes traffic to them based on the health checks.

API Gateway Setup

All the resources we setup so fas are internal to the VPC. To expose the services to the public, we will use the API Gateway. The API Gateway will be configured to use the internal NLB as a VPC link. This private integration will allow the API Gateway to route traffic to the internal services running in the private subnet of the VPC. Requests coming into the API Gateway will be proxied to the internal NLB, which will then route the traffic to the backend services.

To setup teh API gateway and routes, add the following to the cloudformation template:

  # VPC link for REST API and attaching NLB to VPC
  VpcLink:
    Type: "AWS::ApiGateway::VpcLink"
    Properties:
      Name: "appVpcLink"
      Description: "VPC Link for app"
      TargetArns:
        - !Ref LoadBalancer

  # API Gateway
  APIGateway:
    Type: "AWS::ApiGateway::RestApi"
    Properties:
      Name: "appAPI"
      Description: "app API"
      FailOnWarnings: "true"
      EndpointConfiguration:
        Types:
          - "REGIONAL"
      Tags:
        - Key: "Name"
          Value: "appAPI"

  # API Gateway Frontend Resources
  APIGatewayResourceFronetnd:
    Type: "AWS::ApiGateway::Resource"
    Properties:
      RestApiId: !Ref APIGateway
      ParentId: !GetAtt APIGateway.RootResourceId
      PathPart: "frontend"

  # API Gateway Backend Resources
  APIGatewayResourceBackend:
    Type: "AWS::ApiGateway::Resource"
    Properties:
      RestApiId: !Ref APIGateway
      ParentId: !GetAtt APIGateway.RootResourceId
      PathPart: "backend"

  # API Gateway Proxy Resources
  APIGatewayResourceBackendProxy:
    Type: "AWS::ApiGateway::Resource"
    Properties:
      RestApiId: !Ref APIGateway
      ParentId: !Ref APIGatewayResourceBackend
      PathPart: "{proxy+}"

  #Root get method to load the frontend page
  APIGatewayMethodRoot:
    Type: "AWS::ApiGateway::Method"
    Properties:
      RestApiId: !Ref APIGateway
      ResourceId: !GetAtt APIGateway.RootResourceId
      HttpMethod: "GET"
      AuthorizationType: "NONE"
      Integration:
        ConnectionType: "VPC_LINK"
        ConnectionId: !Ref VpcLink
        Type: "HTTP_PROXY"
        IntegrationHttpMethod: "GET"
        Uri: !Sub "http://${LoadBalancer.DNSName}:80"
        IntegrationResponses:
          - StatusCode: 200
      MethodResponses:
        - StatusCode: 200

  # API Gateway Method for Frontend
  APIGatewayMethodFrontend:
    Type: "AWS::ApiGateway::Method"
    Properties:
      RestApiId: !Ref APIGateway
      ResourceId: !Ref APIGatewayResourceFronetnd
      HttpMethod: "GET"
      AuthorizationType: "NONE"
      Integration:
        ConnectionType: "VPC_LINK"
        ConnectionId: !Ref VpcLink
        Type: "HTTP_PROXY"
        IntegrationHttpMethod: "GET"
        Uri: !Sub "http://${LoadBalancer.DNSName}:80"
        IntegrationResponses:
          - StatusCode: 200
      MethodResponses:
        - StatusCode: 200

  # API Gateway Method for Backend
  APIGatewayMethodBackend:
    Type: "AWS::ApiGateway::Method"
    Properties:
      RestApiId: !Ref APIGateway
      ResourceId: !Ref APIGatewayResourceBackendProxy
      HttpMethod: "GET"
      AuthorizationType: "NONE"
      RequestParameters:
        method.request.path.proxy: true
      Integration:
        ConnectionType: "VPC_LINK"
        ConnectionId: !Ref VpcLink
        Type: "HTTP_PROXY"
        IntegrationHttpMethod: "GET"
        Uri: !Sub "http://${LoadBalancer.DNSName}:8000/{proxy}"
        RequestParameters:
          integration.request.path.proxy: "method.request.path.proxy"
        IntegrationResponses:
          - StatusCode: 200
      MethodResponses:
        - StatusCode: 200

  # API Gateway Deployment
  APIGatewayDeployment:
    Type: "AWS::ApiGateway::Deployment"
    Properties:
      RestApiId: !Ref APIGateway
      StageName: "prod"
    DependsOn: [APIGatewayMethodFrontend, APIGatewayMethodBackend]

Here we create two resources, one for the frontend and one for the backend. We then create methods for each resource that will route the traffic to the internal NLB. The NLB will then route the traffic to the backend services.

You can see these resources and methods in the console like this:

API Gateway in AWS Console

The deployment API can be seen under the stages section of the API Gateway. It would look something like this: https://3al3x2812j.execute-api.us-east-2.amazonaws.com/prod

You can invoke the frontend HTML page with the following URL: https://3al3x2812j.execute-api.us-east-2.amazonaws.com/prod/frontend. The backend STATUS API can be invoked with the following URL: https://3al3x2812j.execute-api.us-east-2.amazonaws.com/prod/backend/status

Note: We have not updated the frontend HTML page to use the API resource paths. This is something that can be done in the deployment pipeline using GitHub actions in the following sections.

Scaling and Monitoring

We did not setup CPU load based scaling for the ECS tasks. This can be done by setting up a CloudWatch alarm that triggers an ECS service scaling policy. The scaling policy can be configured to scale up or down based on the CPU load of the ECS tasks.

The ECS tasks can be monitored using CloudWatch logs and metrics. The logs can be viewed in the CloudWatch console, and the metrics can be used to set up alarms and dashboards to monitor the performance of the ECS tasks.

Demand based scaling is a feature of the ECS service that can be used to automatically scale the ECS tasks based on the demand from the backend services. The ECS service will monitor the demand and scale up or down the tasks based on the demand. We are not going into the details of this setup for the time being.

To manually add capacity to the infrastructure, you can update the desired count of the ECS service. This will bring up more tasks in the ECS cluster to handle the load. In the template we setup, the infrastructure is limited to scale up to a maximum of 10 instances. This can be increased based on the requirements.

GitHub Actions for CI/CD

Now that the infrastructure is setup, it is desirable to automate the deployment process in the case of a code change in the application. We will use GitHub actions to automate the deployment process.

I am setting up the repository at https://github.com/embeddedinn/aws_production_lb to host the code and setup the actions. I will also commit the infrastructure code to this repository.

Authentication

Follow the instructions for Configuring OpenID Connect in Amazon Web Services to setup the authentication for the GitHub actions.

The openID connect setup can be done with the following in the cloudformation template:

# OpenID Connect for GitHub
AppGithubOpenIDConnect:
  Type: "AWS::IAM::OIDCProvider"
  Properties:
    Url: "https://token.actions.githubusercontent.com"
    ClientIdList:
      - sts.amazonaws.com # since we will be using the official GitHub actions

A role for this can be created with the following entry in the cloudformation template:

  # IAM Role for GitHub
  AppGithubRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "appGithubRole"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Federated: !GetAtt AppGithubOpenIDConnect.Arn
            Action:
              - "sts:AssumeRoleWithWebIdentity"
            Condition:
              StringEquals:
                # allow only a soecific GitHub account, repo and branch
                token.actions.githubusercontent.com:sub: "repo:embeddedinn/aws_production_lb:ref:refs/heads/main"
                token.actions.githubusercontent.com:aud: "sts.amazonaws.com"
      Policies:
        - PolicyName: "AppGithubPolicy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: "ECSPermissions"
                Effect: "Allow"
                Action:
                  - "ecs:UpdateCluster"
                  - "ecs:ListAttributes"
                  - "ecs:StartTask"
                  - "ecs:DescribeTaskSets"
                  - "ecs:UpdateService"
                  - "ecs:CreateService"
                  - "ecs:RunTask"
                  - "ecs:StopTask"
                  - "ecs:DescribeServices"
                  - "ecs:DescribeTasks"
                  - "ecs:UpdateTaskSet"
                  - "ecs:GetTaskProtection"
                  - "ecs:DeleteService"
                  - "ecs:DescribeClusters"
                  - "ecs:ListTagsForResource"
                  - "ecs:ListContainerInstances"
                  - "ecs:RegisterTaskDefinition"
                  - "ecs:DescribeTaskDefinition"
                Resource:
                  - !Sub "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:cluster/appCluster"
                  - !Sub "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task-definition/appTaskDefinition/*"
                  - !Sub "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task/appCluster/*"
                  - !Sub "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task-set/appCluster/*"
                  - !Sub "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:service/appCluster/*"
                  - !Sub "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:container-instance/appCluster/*"

              - Sid: "ECRPermissions"
                Effect: "Allow"
                Action:
                  - "ecr:StartImageScan"
                  - "ecr:DescribeImageReplicationStatus"
                  - "ecr:ListTagsForResource"
                  - "ecr:UploadLayerPart"
                  - "ecr:BatchGetRepositoryScanningConfiguration"
                  - "ecr:ListImages"
                  - "ecr:TagResource"
                  - "ecr:DescribeRepositories"
                  - "ecr:CompleteLayerUpload"
                  - "ecr:BatchCheckLayerAvailability"
                  - "ecr:GetLifecyclePolicy"
                  - "ecr:PutLifecyclePolicy"
                  - "ecr:DescribeImageScanFindings"
                  - "ecr:GetDownloadUrlForLayer"
                  - "ecr:DeleteLifecyclePolicy"
                  - "ecr:PutImage"
                  - "ecr:UntagResource"
                  - "ecr:BatchGetImage"
                  - "ecr:DescribeImages"
                  - "ecr:StartLifecyclePolicyPreview"
                  - "ecr:InitiateLayerUpload"
                Resource:
                  - !Sub "arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ECRRepository}"
                  - !Sub "arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ECRRepository}:image/*"

              - Sid: "GithubActionsPermissions"
                Effect: "Allow"
                Action:
                  - "ecr:GetAuthorizationToken"
                  - "ecs:DescribeTaskDefinition"
                Resource: "*"

              - Sid: "VisualEditor3"
                Effect: "Allow"
                Action:
                  - "iam:PassRole"
                Resource:
                  - !Sub "arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole"
                  - !Sub "arn:aws:iam::${AWS::AccountId}:role/ecsServiceRole"
                  - !Sub "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"
                  - !Sub "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs-tasks.amazonaws.com/AWSServiceRoleForECSTasks"
                  - !Sub "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"

Now, setup a GitHub action to build and push the docker image to the ECR repository. The following is the GitHub action that can be used:

name: Push-to-ecr-deploy-ecs

on:
#  push:
#    branches: [ "main" ]
#  pull_request:
#    branches: [ "main" ]
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v2
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v3
        with:
          role-to-assume: arn:aws:iam::338186951935:role/appGithubRole
          aws-region: us-east-2

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Download task definition
        run: aws ecs describe-task-definition --task-definition appTaskDefinition --query taskDefinition > task-definition.json

      - name: Build, tag, and push docker image to Amazon ECR
        id: build-image
        env:
          REGISTRY: $
          REPOSITORY: app_repository
          IMAGE_TAG: $
        run: |
          cd src
          docker build -t $REGISTRY/$REPOSITORY:$IMAGE_TAG .
          docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG
          echo "image=$REGISTRY/$REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT

      - name: Fill in the new image ID in the Amazon ECS task definition
        id: task-def
        uses: aws-actions/amazon-ecs-render-task-definition@v1
        with:
          task-definition: task-definition.json
          container-name: appContainer
          image: $

      - name: Deploy Amazon ECS task definition
        uses: aws-actions/amazon-ecs-deploy-task-definition@v2
        with:
          task-definition: $
          service: appService
          cluster: appCluster
          wait-for-service-stability: true

Now, once we update the app and commit the changes to Github, we can trigger the action to build and deploy the app to the ECS cluster. Existing tasks will be replaced with the new tasks with the updated image. Since we chose a t2.micro instance that might not be able to host two tasks, we set the MinimumHealthyPercent of the service to 50. This will ensure that at least one task is running at all times, and lets the deployment process take down one task at a time to replace it.

Tracking resource outputs

The relevant outputs from the cloudformation stack can be tracked using the following command:

# Outputs
Outputs:
  VPCId:
    Description: "VPC ID"
    Value: !Ref VPC

  PublicSubnetId:
    Description: "Public Subnet ID"
    Value: !Ref PublicSubnet

  PrivateSubnetId:
    Description: "Private Subnet ID"
    Value: !Ref PrivateSubnet

  LoadBalancerDNSName:
    Description: "DNS name of the Load Balancer"
    Value: !GetAtt LoadBalancer.DNSName

  ECRRepositoryURI:
    Description: "URI of the ECR Repository"
    Value: !GetAtt ECRRepository.RepositoryUri

  APIGatewayURL:
    Description: "URL of the API Gateway"
    Value: !Sub "https://${APIGateway}.execute-api.${AWS::Region}.amazonaws.com/prod/"

  AppGithubRoleARN:
    Description: "ARN of the IAM Role for GitHub"
    Value: !GetAtt AppGithubRole.Arn

  TaskDefinition:
    Description: "ARN of the ECS Task Definition Name"
    Value: !Ref ECSTaskDefinition

Conclusion

In this post, we setup a production ready infrastructure in AWS using CloudFormation. We setup a VPC with public and private subnets, an internal NLB, an ECS cluster, an API Gateway, and a GitHub action to automate the deployment process. We also setup the capacity provisioning, load balancer, and target groups to route traffic to the backend services.

The code and templates used in this article can be found in github at embeddedinn/aws_production_lb.

Overview

Consider a scenario where a device equipped with a sensor provides data encapsulated within a C structure. The interface to access this data might utilize buses ranging from simple ones like UART or SPI to more complex ones like USB or PCI. When such a device connects to a host (e.g., a PC), the host typically leverages a C library to retrieve sensor data. This C library not only provides APIs to access the sensor but also interfaces with the host’s low-level OS functions to facilitate communication over the designated bus.

Once the data is available on the host, processing often requires a high-level programming language like Python that is easier to work with.

The primary challenge lies in seamlessly interfacing the C library with Python, especially when dealing with complex data structures. This is where ctypes and SWIG come into play.

ctypes

ctypes is Python’s Foreign Function Interface (FFI) library, enabling direct interaction with C libraries. It offers C-compatible data types and allows Python code to call functions within DLLs or shared libraries. Beyond calling functions, ctypes facilitates passing simple data types and pointers to data structures between C and Python.

Being a part of the Python standard library, ctypes is universally available across platforms supporting Python, making it a robust choice for interfacing with C libraries.

SWIG

SWIG, an acronym for Simplified Wrapper and Interface Generator, is a powerful tool that connects C and C++ programs with a multitude of high-level programming languages. It’s compatible with scripting languages such as Perl, PHP, Python, Tcl, Ruby, and PHP, as well as non-scripting languages like C#, Java, Lua, Modula-3, OCAML, Octave, and R.

SWIG stands out when dealing with complex data structures. It not only generates the necessary Python code to interface with C but also creates the requisite shared libraries. Its capability to simplify and maintain interfaces between C code and Python makes it an invaluable tool in the developer’s arsenal.

ctypes Example

Let’s start with a simple example of accessing a C data structure from Python using ctypes. Consider the following C code:

// point.c
#include <stdio.h>

typedef struct {
    int x;
    int y;
} Point;

void print_point(Point *p) {
    printf("Point: (%d, %d)\n", p->x, p->y);
}

This code defines a simple Point structure with two integer fields, x and y, and a function print_point that prints the coordinates.

To build this into a shared library:

gcc -shared -o libpoint.so -fPIC point.c

The -fPIC flag ensures position-independent code generation, essential for shared libraries, while the -shared flag specifies the creation of a shared library.

In Python, using ctypes to interact with this library:

import ctypes

class Point(ctypes.Structure):
    _fields_ = [("x", ctypes.c_int),
                ("y", ctypes.c_int)]

lib = ctypes.CDLL("./libpoint.so")
lib.print_point.argtypes = [ctypes.POINTER(Point)]
lib.print_point.restype = None

p = Point(10, 20)
lib.print_point(ctypes.byref(p))

Here, we define the Point class mirroring the C structure. The shared library is loaded, and the print_point function’s argument and return types are specified. Finally, a Point instance is created and passed to the C function.

Handling More Complex Structures with ctypes

When dealing with intricate data structures, direct interaction via ctypes becomes more involved. Let’s enhance our C code:

// point.c
#include <stdio.h>
#include <stdlib.h>

typedef struct {
    int x;
    int y;
    struct {
        int a;
        int b;
    } z;
    union {
        int c;
        int d;
    } w;
} Point;

Point *create_point(int x, int y, int a, int b, int c) {
    Point *p = (Point *)malloc(sizeof(Point));
    p->x = x;
    p->y = y;
    p->z.a = a;
    p->z.b = b;
    p->w.c = c;
    return p;
}

void update_point(Point *p, int x, int y, int a, int b, int c) {
    p->x = x;
    p->y = y;
    p->z.a = a;
    p->z.b = b;
    p->w.c = c;
}

We build this into a shared library using the following command:

gcc -shared -o libpoint.so -fPIC point.c

After building the shared library similarly as before, the corresponding Python code using ctypes would be:

import ctypes

class Point(ctypes.Structure):
    _fields_ = [("x", ctypes.c_int),
                ("y", ctypes.c_int),
                ("z", ctypes.c_int * 2),
                ("w", ctypes.c_int)]

lib = ctypes.CDLL("./libpoint.so")
lib.create_point.argtypes = [ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int]
lib.create_point.restype = ctypes.POINTER(Point)

lib.update_point.argtypes = [ctypes.POINTER(Point), ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int]
lib.update_point.restype = None

p = lib.create_point(10, 20, 30, 40, 50)
print("Point: (%d, %d, %d, %d, %d)" % (p.contents.x, p.contents.y, p.contents.z[0], p.contents.z[1], p.contents.w))

lib.update_point(p, 100, 200, 300, 400, 500)
print("Point: (%d, %d, %d, %d, %d)" % (p.contents.x, p.contents.y, p.contents.z[0], p.contents.z[1], p.contents.w))

In this code, we define a more complex Point structure with nested structures and unions. We define two functions in the C code: create_point that allocates and initializes a Point structure and update_point that updates the fields of a Point structure. We then access these functions from Python using ctypes.

SWIG Example

While ctypes is powerful, redefining complex structures in Python can be tedious and error-prone. SWIG streamlines this process by auto-generating the necessary Python wrappers.

First, let’s externalize our data structure definition into a header file, point.h:

// point.h
#ifndef POINT_H
#define POINT_H

typedef struct {
    int x;
    int y;
    struct {
        int a;
        int b;
    } z;
    union {
        int c;
        int d;
    } w;
} Point;

Point *create_point(int x, int y, int a, int b, int c);
void update_point(Point *p, int x, int y, int a, int b, int c);

#endif

Next, we create a SWIG interface file point.i. This interface definition can be done directly in the headerfile as well. However, we will keep it separate for extensibility. The interface file defines the functions that need to be exposed to Python:

%module point

%{
#include "point.h"
%}

%include "point.h"

This interface file exposes all the functions and data structures defined in the point.h file.

The next step requires SWIG to be installed on the system. SWIG can be installed using the following command:

sudo apt install swig

We can now generate the Python code and the shared library using the following command:

swig -python point.i

This step generates a point_wrap.c file that contains the interface code that be used to generate the shared library. It also generates the point.py file that contains the Python code that interfaces with the shared library.

To build the shared library, we use the following command:

gcc -shared -o _point.so -fPIC point.c point_wrap.c -I/usr/include/python3.11

Note that the -I flag is used to specify the path to the Python header files. You need the python development files installed on your system to build the shared library. You can install it with

sudo apt install python3-dev

The shared library is named _point.so because it is a Python extension module. It is linked to the point.c file and the point_wrap.c file generated by SWIG.

Now, we can import the point module in Python and access the functions and data structures defined in the C code:

import point

p = point.create_point(10, 20, 30, 40, 50)
print("Point: (%d, %d, %d, %d, %d)" % (p.x, p.y, p.z.a, p.z.b, p.w.c))

point.update_point(p, 100, 200, 300, 400, 500)
print("Point: (%d, %d, %d, %d, %d)" % (p.x, p.y, p.z.a, p.z.b, p.w.d))

In this code, we import the point module generated by SWIG and access the create_point and update_point functions. We also access the fields of the Point structure directly.

As we can see, SWIG simplifies the process of interfacing C code with Python and makes it easier to maintain the code. It also generates the Python code that interfaces with the C code and the shared library that can be used by Python.

Advanced SWIG

We can now look at some advanced features of SWIG that can be used to implement some complex use cases. SWIG provides a lot of features that can be used to customize the interface between C and Python.

Typemaps

Typemaps are used to convert data between C and Python. SWIG provides a lot of built-in typemaps that can be used to convert data between C and Python. Typemaps can be used to convert simple data types, pointers, arrays, structures, and classes between C and Python.

For example, consider the following C code:

// point.h
#ifndef POINT_H
#define POINT_H

typedef struct {
    int x;
    int y;
} Point;

Point *create_point(int x, int y);
int update_point(Point *p, Point data);

#endif

// point.c
#include <stdio.h>
#include <stdlib.h>
#include "point.h"

Point *create_point(int x, int y) {
    Point *p = (Point *)malloc(sizeof(Point));
    p->x = x;
    p->y = y;
    return p;
}

int update_point(Point *p, Point data) {
    p->x = data.x;
    p->y = data.y;
    return 0;
}

We can define a typemap in the SWIG interface file to convert the Point structure to a Python tuple and vice versa in the point.i file:

%module point

%{
#include "point.h"
%}

%include "point.h"

%typemap(in) Point {
    $1 = (Point *)malloc(sizeof(Point));
    $1->x = PyLong_AsLong(PyTuple_GetItem($input, 0));
    $1->y = PyLong_AsLong(PyTuple_GetItem($input, 1));
}

%typemap(argout) Point {
    $result = PyTuple_Pack(2, PyLong_FromLong($1->x), PyLong_FromLong($1->y));
}

%typemap(freearg) Point {
    free($1);
}

In this code, we define an in typemap that converts a Python tuple to a Point structure. We also define an argout typemap that converts a Point structure to a Python tuple. We also define a freearg typemap that frees the memory allocated for the Point structure.

Now, regenerate the interface code and the shared library using SWIG and the following commands:

swig -python point.i
gcc -shared -o _point.so -fPIC point.c point_wrap.c -I/usr/include/python3.11

We test this with the following Python code:

import point

p = point.create_point(10, 20)
print("Point: (%d, %d)" % (p.x, p.y))

p1 = point.create_point(30, 40)
print("Point: (%d, %d)" % (p1.x, p1.y))

point.update_point(p, p1)
print("Point: (%d, %d)" % (p.x, p.y))

creating transmittable data structure blobs

In some cases, we may need to transmit the data structure over a network or save it to a file. We can create a blob that contains the data structure and then transmit it over the network or save it to a file. We can use the struct module in Python to create a blob that contains the data structure.

An example of this use case is to generate configuration data stored in the device memory based on user inputs. The tool to generate the configuration data can be written in Python and the configuration data can be transmitted to the device over a configuration interface like UART or SPI. A SWIG based interface can be used to ensure that the configuration data is correctly formatted and transmitted to the device.

The configuration data-structure can be defined in C as follows:

// config.h
#ifndef CONFIG_H
#define CONFIG_H

#include <stdint.h>

#pragma pack(1)

typedef struct {
        uint8_t i2cAddress;
        uint32_t i2cCtrlConfig[2];
    }  i2cConfig_t;

typedef struct {
    uint32_t config1;
    uint32_t ledConfig;
    i2cConfig_t i2cConfig;
    char deviceName[256];
} config_t;

#endif

Note that this structure contains a “complex data structure” that swig might not be able to handle directly. We can use some inline functions and extensions to handle this.

The SWIG interface file config.i can be defined as follows:

%module config

%{
#include "config.h"
%}

%include "config.h"
%include "stdint.i"


%inline %{
// inline function to set device name
void setDeviceName(config_t *configData, const char *name) {
    //error if name is longer than 256
    if(strlen(name) > 256) {
        fprintf(stderr, "Device name is too long\n");
        return;
    }
    strncpy(configData->deviceName, (char*)name, 256);
}

// inline function to get device name
const char *getDeviceName(config_t *configData) {
    return configData->deviceName;
}

%}

%extend i2cConfig_t {
    void setI2cCtrlConfig(int index, uint32_t value) {
        $self->i2cCtrlConfig[index] = value;
    }
    uint32_t getI2cCtrlConfig(int index) {
        return $self->i2cCtrlConfig[index];
    }
}

//implement a serialization function in c to serialize the config_t struct to a file
%extend config_t {
    int serialize(const char *filename) {
        FILE *f = fopen(filename, "wb");
        int len=fwrite($self, sizeof(config_t), 1, f);
        fclose(f);
        return len*sizeof(config_t);
    }
}

The Python code to generate the configuration data blob can be written as follows:

#create.py
import config

configData = config.config_t()
configData.config1 = 0x12345678
configData.ledConfig = 0x87654321
config.setDeviceName(configData, "MyDevice")

i2cConfig = config.i2cConfig_t()
i2cConfig.i2cAddress = 0x50
i2cConfig.setI2cCtrlConfig(0, 0x12345678)
i2cConfig.setI2cCtrlConfig(1, 0x87654321)
configData.i2cConfig = i2cConfig

#print data being written to file in hex

print(f"config1: {hex(configData.config1)}")
print(f"ledConfig: {hex(configData.ledConfig)}")
print(f"deviceName: {config.getDeviceName(configData)}")
print(f"i2cAddress: {hex(configData.i2cConfig.i2cAddress)}")
print(f"i2cCtrlConfig0: {hex(configData.i2cConfig.getI2cCtrlConfig(0))}")
print(f"i2cCtrlConfig1: {hex(configData.i2cConfig.getI2cCtrlConfig(1))}")


fileLen=configData.serialize("config.bin")
print("File length: ", fileLen)

In this code we define a configData object of type config_t and set its fields. We also define an i2cConfig object of type i2cConfig_t and set its fields. We then set the i2cConfig object as a field of the configData object. We then print the contents of the configData object and save it to a file using the serialize function.

NOTE: the code aligns the data to 1 byte boundaries using #pragma pack(1). This is important as the data structure may be padded by the compiler to align it to 4 byte boundaries. Make sure that you update this to match the alignment of the data structure in your C code.

build the interface and run the code to create the configuration binary with :

swig -python config.i
gcc -shared -o _config.so -fPIC config_wrap.c -I/usr/include/python3.11
python3 create.py

We then generate the configuration data blob in Python and save it to a file. The configuration data blob can then be transmitted over the network or saved to a file. To read this in C, we can use the following code:

//read_config.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "config.h"

int main() {
    config_t *configData = (config_t *)malloc(sizeof(config_t));
    FILE *f = fopen("config.bin", "rb");
    fread(configData, sizeof(config_t), 1, f);
    fclose(f);

    printf("config1: 0x%x\n", configData->config1);
    printf("ledConfig: 0x%x\n", configData->ledConfig);
    printf("deviceName: %s\n", configData->deviceName);
    printf("i2cAddress: 0x%x\n", configData->i2cConfig.i2cAddress);
    printf("i2cCtrlConfig[0]: 0x%x\n", configData->i2cConfig.i2cCtrlConfig[0]);
    printf("i2cCtrlConfig[1]: 0x%x\n", configData->i2cConfig.i2cCtrlConfig[1]);
    
    free(configData);
    return 0;
}

We can build and execute the C code using the following command:

gcc -o read_config read_config.c
./read_config

NOTE: This example assumes that the systems executing the python code and the C code have similar endianess and data alignment requirements. If the systems have different architectural requirements additional processing or conversion may be required.

Packaging the Python code

We can package the interface code and the shared library into a Python package for easy distribution. We will explore two methods - one that involves a build process at install time and one that involves a build process at package creation time.

Performing build at install time lets the interface be compiled on the target system. This is useful when the target system may have different architectures or requirements. The package can be installed on the target system and the interface can be compiled during the installation process.

### Building the interface in the target system at install time

To package the Python code, we need to create a setup.py file that contains the package information and the build instructions. The setup.py file can be defined as follows:

# setup.py
from setuptools import setup, Extension

setup(
    name='config',
    version='1.0',
    ext_modules=[Extension('_config', ['config.i', 'config.c'])],
    py_modules=["config"],
)

In this code, we define the config package with the config module. We also define the shared library _config with the config.i and config.c files. The config.i file contains the SWIG interface definition and the config.c file contains the C code.

We can now build the package using the following command:

python3 setup.py build

The python3 setup.py build command will build the shared library and the Python module. The shared library will be built in the build/lib.linux-x86_64-3.11 directory and the Python module will be built in the build/lib directory. The SWIG command will be executed during the build process to generate the interface code as setuptools knows how to handle the .i files mentioned in the Extension object.

The package can be installed using the following command:

python3 setup.py install

NOTE: It is adviced that the package be installed in a virtual environment to avoid conflicts with system packages. The package can also be installed using the –user flag to install it in the user’s home directory.

This package can now be distributed and installed on any system. To do this with a package manager like pip, we can create a source distribution using the following command:

python3 setup.py sdist

This command will create a dist directory with a .tar.gz file that contains the package. The package can be installed using the following command:

pip install dist/config-1.0.tar.gz

Building the interface in the build system at package creation time

We can use the same setup.py file with the bist_wheel command to build a wheel package that contains the shared library and the Python module. The wheel package can be installed on any system without the need to compile the shared library.

python3 setup.py bdist_wheel

Note: Make sure that the system has the wheel package installed. This can be installed with pip install wheel

This step creates a dist directory with a .whl file that contains the package. The package can be installed using the following command:

pip install dist/config-1.0-py3-none-any.whl

This package can now be distributed and installed on any system without the need to compile the shared library. This is useful when the target system may not have the required build tools or dependencies to compile the shared library.

Conclusion

In this post, we explored how to access complex C data structures from Python using ctypes and SWIG. We also explored how to pass complex data structures between C and Python and perform a few operations on them. The Python infrastructure was also packaged as a shared library / Package for easy distribution. We also explored two methods of packaging the Python code - one that involves a build process at install time and one that involves a build process at package creation time.

FreeRTOS is a popular real-time operating system that is widely used in embedded systems. It is open-source and has a large user base. It is known for its small footprint and ease of use. In this post, we will explore how to setup a development environment for FreeRTOS on RISCV using QEMU. this includes setting up the RISCV toolchain, QEMU, and building and executing FreeRTOS examples. We will also explore some techniques to debug FreeRTOS applications running on QEMU.

Setting up the Environment

Toolchain

To get started, we need to set up a RISCV toolchain and QEMU. We will be building the Toolchain from source. The steps to build the toolchain are as follows:

1. Install dependencies. (I am using debian based system, so the commands might be different for other systems)

    sudo apt-get install autoconf automake autotools-dev curl python3 python3-pip libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev libexpat-dev ninja-build git cmake libglib2.0-dev libslirp-dev
   

2. Setup the installation directory for the toolchain:

    sudo mkdir -p /opt/riscv
    sudo chown $USER:$USER /opt/riscv
   

3. Clone the RISCV toolchain repository from GitHub:

    git clone https://github.com/riscv/riscv-gnu-toolchain
   

4. Build and install the multilib toolchain:

    cd riscv-gnu-toolchain
    ./configure --prefix=/opt/riscv --enable-multilib
    make -j$(nproc)
   

   `

5. Add the toolchain to your PATH:

    export PATH=$PATH:/opt/riscv/bin
   

6. Verify the installation:

    which riscv64-unknown-elf-gcc
   

   You should see the path to the toolchain binary. You can also verify the version of the toolchain using the following command:

    riscv64-unknown-elf-gcc --version
   

   At the time of writing, this is the output I get:

    riscv64-unknown-elf-gcc (gc891d8dc23e) 13.2.0
    Copyright (C) 2023 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
   

Qemu

Next, we need to install QEMU. We will be building QEMU from source in order to get the latest . The steps to build QEMU are as follows:

1. install dependencies:

    sudo apt-get install libglib2.0-dev libpixman-1-dev
   

2. Setup the installation directory for QEMU:

    sudo mkdir -p /opt/qemu
    sudo chown $USER:$USER /opt/qemu
   

3. Clone the QEMU repository from GitHub:

    git clone https://git.qemu.org/git/qemu.git
   

4. Build and install QEMU:

    cd qemu
    ./configure --prefix=/opt/qemu
    make -j$(nproc)
    make install
   

5. Export the QEMU path:

    export PATH=$PATH:/opt/qemu/bin
   

6. Verify the installation:

    which qemu-system-riscv64
   

   You should see the path to the QEMU binary. You can also verify the version of QEMU using the following command:

    qemu-system-riscv64 --version
   

   At the time of writing, this is the output I get:

    QEMU emulator version 9.0.50 (v9.0.0-1157-g121e47c8bf)
    Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
   

FreeRTOS

Next, we need to clone the FreeRTOS repository from GitHub:

```bash
git config --global core.symlinks true
git clone https://github.com/FreeRTOS/FreeRTOS.git --recurse-submodules
```

Building and executing FreeRTOS examples

FreeRTOS provides a set of examples that can be used to understand the working of the OS. We will be using the example under `FreeRTOS/Demo/RISC-V_RV32_QEMU_VIRT_GCC` as a starting point to build and execute FreeRTOS for RISCV on QEMU. 

The example includes a readme file that provides instructions on how to build the example. The key steps are as follows:

1. Build the example:

    cd FreeRTOS/Demo/RISC-V_RV32_QEMU_VIRT_GCC
    make -C build/gcc DEBUG=1
   

   This generates the output binary make -C build/gcc

2. Execute the binary using QEMU:

    qemu-system-riscv32 -nographic -machine virt -net none \
    -chardev stdio,id=con,mux=on -serial chardev:con \
    -mon chardev=con,mode=readline -bios none \
    -smp 4 -kernel ./build/gcc/output/RTOSDemo.elf
   

   The output looks like this:

    FreeRTOS Demo Start
    FreeRTOS Demo SUCCESS: : 5033
    FreeRTOS Demo SUCCESS: : 10034
    FreeRTOS Demo SUCCESS: : 15033
    FreeRTOS Demo SUCCESS: : 20034
    FreeRTOS Demo SUCCESS: : 25033
    FreeRTOS Demo SUCCESS: : 30034
    FreeRTOS Demo SUCCESS: : 35033
    FreeRTOS Demo SUCCESS: : 40033
    FreeRTOS Demo SUCCESS: : 45034
    FreeRTOS Demo SUCCESS: : 50033
    FreeRTOS Demo SUCCESS: : 55034
    FreeRTOS Demo SUCCESS: : 60033
    FreeRTOS Demo SUCCESS: : 65033
    FreeRTOS Demo SUCCESS: : 70033
    FreeRTOS Demo SUCCESS: : 75033
    FreeRTOS Demo SUCCESS: : 80033
    FreeRTOS Demo SUCCESS: : 85033
   

Debugging the application

We will use the GDB capabilities of QEMU and VScode to debug the application. To do this, we need to start QEMU with the GDB server enabled. The steps are as follows:

1. Start QEMU with the GDB server enabled by adding the -s flag. -S ensures that the CPU is halted until GDB connects to it. Here is the command to start QEMU:

    qemu-system-riscv32 -nographic -machine virt -net none \
    -chardev stdio,id=con,mux=on -serial chardev:con \
    -mon chardev=con,mode=readline -bios none \
    -smp 4 -kernel ./build/gcc/output/RTOSDemo.elf -s -S
   

2. Start GDB and connect to QEMU. We will test the commandline GDB first, before moving to VScode. Here is the command to start GDB:

    # start GDB and connect to QEMU on port 1234
    riscv64-unknown-elf-gdb ./build/gcc/output/RTOSDemo.elf -ex "target remote :1234"
   

To start executing the code, type c in the GDB prompt. You can set breakpoints and step through the code using the GDB commands.

Debugging using VScode

1. Install the native-debug extension in VScode from https://marketplace.visualstudio.com/items?itemName=webfreak.debug

2. Open the debug pannel (Ctrl+Sift+D) and create a launch.json file with the following contents:

    {
    "configurations": [
            {
                "type": "gdb",
                "request": "attach",
                "name": "Attach to gdbserver",
                "executable": "${workspaceFolder}/build/gcc/output/RTOSDemo.elf",
                "target": ":1234",
                "remote": true,
                "cwd": "${workspaceRoot}",
                "gdbpath": "/opt/riscv/bin/riscv64-unknown-elf-gdb",
                "valuesFormatting": "parseText"
            }
        ]
    }
   

3. Click on the “Run and Debug” button and select the “Attach to gdbserver” configuration. This will start the debugging session. In the image below, I have put a breakpoint in main(). The callstack and registers are visible in the debug window.

Debug window

Conclusion

In this post, we explored how to set up a development environment for FreeRTOS on RISCV using QEMU. We built the RISCV toolchain and QEMU from source and executed a FreeRTOS example on QEMU. We also explored how to debug the application using GDB and VScode. This setup can be used to develop and debug FreeRTOS applications on RISCV using QEMU. In future posts, we will explore more of FreeRTOS on RISCV. Stay tuned!

This is not one of my typical technical deep dives. This is a post about why I blog, based on a conversation I had with my nephew. I hope you find it interesting.

I recently had a long chat with my nephew, who was about to start his Bachelor’s in Engineering. We discussed the advice I wish I had received when I was in his shoes. One key piece of learning that emerged was the value of documenting what you learn. My nephew, like many, viewed writing as a purely theoretical exercise, detached from the practical world of Engineering. I understood his perspective, as I, too, held this belief in my youth. However, I realized that writing about my learning experiences was a crucial tool for effective learning. This post delves into why this practice is essential and how it has personally benefited me.

I have always been an active learner. I need to read a lot, watch videos, and do a lot of practical work. It is difficult for my brain to internalize concepts by just reading or watching. I need to do something with the knowledge to understand it. So, for the better part of my college and early career years, I always found projects to work on and built some cool stuff. But I never wrote about what I learned. I would write code, build circuits, do a lot of practical work and move on to the next project. Over time, I realized that while the core concepts were clear, I could have articulated them better. I struggled to explain the concepts to others, and I often needed to remember details of the projects I had worked on. I realized that I was retaining the knowledge I gained at a macro level, but I was sometimes unable to recall the details. More importantly, though I was touching upon many exciting concepts, I was learning enough to finish the work. I needed to learn to apply the concepts’ broader aspects to other problems. I often found myself in a situation where I would have to relearn concepts I had already learned and apply them to new problems.

Today’s learning happens through YouTube videos, online courses and even AI assistants. But, when I was getting started, google searches about niche topics often led to simple HTML pages written by enthusiasts. These pages are very detailed and straightforwardly explain the concepts. I often found myself going back to these pages to refer to the concepts and realizing that I would learn something new each time I went back. This meant I could only internalize the concepts partially the first time I read them. I realized that I needed to be learning more effectively. This was humbling and inspiring at the same time. This is what led me to start writing about what I learned.

As a start, I wrote about something I (thought I) knew well at the time - PIC 8 Microcontrollers. During my university days, I spent considerable time writing complex assembly code for PIC8 microcontrollers, and I thought I knew the architecture well. I wrote a blog post about the architecture and the instruction set. I was surprised that I had forgotten a few concepts and missed others. I realized this only when I tried to explain the concepts to a virtual audience. A few years later, when I started working for Microchip, I learned even more gaps in my understanding, which triggered quite a bit of my imposter syndrome. But that’s a story for another day. I have re-homed the post from its original Google Sites location to embeddedinn.com.

This first experience was eye-opening, and I realized that writing about what I learn is a great way to learn effectively. Since then, whenever I do a professional or hobby project, I try to write about it at embeddedinn.com. Though it started with articles about the projects themselves, I eventually started including a lot of articles about the concepts I learned while working on the projects.

A happy side-effect of writing about what I learned is that it has helped me professionally. I have been able to articulate my thoughts better, and I have been able to explain concepts to my colleagues and customers better. It has also triggered a lot of exciting side conversations with people who read my articles. Some readers have extreme opinions about my take on things, and often, these conversations lead to more exciting projects and learning opportunities.

Today, the list of things I have written about is an excellent resource for me. I often return to my articles to refer to concepts I have written about. Some of these articles have also become inspiration and reference materials for other like-minded people learning about the same concepts. The challenge now is balancing the backlog of items I have to write about from past projects and completing new exciting projects I take on. But it’s a good problem to have.

Introduction

Device trees are typically used in Linux systems to describe the hardware components of a system. Lately, RTOSes like Zephyr have made it more popular among developers of systems with lesser complexity. Device trees can be used in any bare-metal system to describe a system’s hardware components and pass run-time parameters (as opposed to compile-time parameters). This approach offers a great deal of interoperability and portability. It also allows for a clear separation of hardware and software, making it easier to maintain and update the software. This article describes how to use device trees in bare-metal systems.

We will follow the latest 0.4 version of the Device Tree Specification.

Device Tree Overview

A Device tree is a hierarchical data structure often used to describe the hardware components and their configuration in a system. A device tree source (DTS) is a human-readable representation of the tree. The device tree compiler (DTC) is used to convert the DTS to a binary device tree blob (DTB) that can be used by software like a bootloader or the kernel.

Device tree bindings are similar to a schema in XML. It describes the properties and nodes used to describe a hardware component. The bindings are used to validate the device tree source and to generate documentation.

Device tree overlays (dtbo) are used to modify the device tree at runtime. This is useful for adding or removing hardware components from the device tree without modifying the device tree source. For example, when a beaglebone cape is added to the system, the device tree overlay can be used to add the cape to the device tree without modifying the device tree source.

Systems like Zephyr primarily convert device tree sources and bindings to produce a generated C header. However, in some systems, like an embedded system running Linux, the DTB is stored in a known location and is loaded into the kernel by the bootloader.

Device Tree Syntax Basics

Before we get into the details of using device trees in bare-metal systems, let’s review the basics of the device tree syntax.

Properties

A property is a key-value pair that describes a hardware component. A property is defined using the following syntax:

property-name = <value>;

The property name is a unique identifier for the property. The value can be a number, a string, or a list of numbers.

compatible = "vendor,device";
reg = <0x12340000 0x1000>;
interrupts = <0 10 0>;

In this example, the compatible property is a string, the reg property is a list of numbers, and the interrupts property is a list of numbers.

Some of the standard properties from the specification are tabulated below:

Nodes

A node is a collection of properties that describe a hardware component. A node is defined using the following syntax:

node-name {
    property-name = <value>;
    ...
};

The node name is a unique identifier for the node. The properties are a collection of key-value pairs that describe the hardware component. The value can be a number, a string, or a list of numbers.

Paths

A path is a unique identifier for a node in the device tree. It is a collection of node names separated by slashes. For example, the path /soc/uart@12340000 refers to the node with the name uart@12340000, which is a child of the node with the name soc.

Includes

The #include directive includes other device tree sources in the current device tree source. This is useful for reusing common device tree definitions across multiple sources.

#include "common.dtsi"

Comments

Comments in the device tree source are similar to comments in the C programming language. They start with // and continue to the end of the line.

// This is a comment

Example

Practically, a device tree corresponds to a hardware component. For example, consider an embedded system with a UART, a GPIO, and a SPI controller. The SPI controller is connected to a Flash memory, a temperature sensor, and an accelerometer.

The DTS for this system would look something like this:

Note: The following example is a simplified version of the device tree. The device tree would be more complex and include additional properties and nodes.

/dts-v1/;
/ {
    compatible = "vendor,device";
    #address-cells = <1>;
    #size-cells = <1>;

    soc {
        compatible = "vendor,soc";
        #address-cells = <1>;
        #size-cells = <1>;

        uart@800000 {
            compatible = "vendor,uart";
            reg = <0x800000 0x100>;
            interrupts = <10 0>;
        };

        gpio@800100 {
            compatible = "vendor,gpio";
            reg = <0x800100 0x100>;
            interrupts = <11 0>;
        };

        spi@800200 {
            compatible = "vendor,spi";
            reg = <0x800200 0x100>;
            interrupts = <12 0>;
            #address-cells = <1>;
            #size-cells = <1>;

            flash@0 {
                compatible = "vendor,flash";
                chip-select = <0>;
		        flash-size=<0x100000>;
            };

            temp-sensor@1 {
                compatible = "vendor,temp-sensor";
                chip-select = <1>;
            };

            accel@2 {
                compatible = "vendor,accel";
                chip-select = <2>;
            };
        };
    };
};

The idea is to change the hardware configuration without changing the software. This is especially useful in systems with many hardware components and systems that are expected to be used in different configurations. For example, we should be able to use the same software on a system with the chip select lines for the Flash and Temperature sensor swapped without changing the software.

To make this possible, the Flash and Temperature sensor device driver should not hardcode the chip select lines. Instead, it should use the device tree to find the chip select lines. This way, the device driver can be used on different systems without modification. The device tree will be loaded from a known location in the system, and the device driver will use the device tree to find the chip select lines.

Using Device Trees in Bare-Metal Systems

First, we must compile the device tree into a binary device tree blob (DTB). Store the device tree from the example above in a file called soc.dts. Then, use the device tree compiler to compile the device tree into a binary device tree blob:

dtc -I dts -O dtb -o soc.dtb soc.dts

Since this is a simplified, non-standard example, you will see warnings about missing properties and nodes. You can safely ignore these warnings for now.

Note dtc can also decompile a device tree blob into a device tree source. This is useful for debugging and understanding how the device tree compiler works. To decompile a device tree blob, use the -I dtb and -O dts options to specify the input and output formats, respectively.

The output of the device tree compiler is a binary device tree blob, often called the flat device tree (FDT). It is referred to as a flat device tree because it is a flat memory image that contains the entire device tree that also brings in the includes.

The Devicetree .dtb Structure is depicted in the following diagram:

The fdt_header is a fixed-size header that contains information about the device tree. The layout of the header for the device tree is defined by the following C structure.

struct fdt_header {
  uint32_t magic;              //0xd00dfeed
  uint32_t totalsize;          //total size of the device tree in bytes
  uint32_t off_dt_struct;      //offset in bytes of the structure block
  uint32_t off_dt_strings;     //offset in bytes of the strings block
  uint32_t off_mem_rsvmap;     //offset in bytes of the memory reservation block
  uint32_t version;            //format version
  uint32_t last_comp_version;  //last compatible version
  uint32_t boot_cpuid_phys;    //physical ID of the boot CPU. Not applicable in non-standard systems
  uint32_t size_dt_strings;   //size of the strings block in bytes
  uint32_t size_dt_struct;    //size of the structure block in bytes
};

Please refer to the Device Tree Specification for a full description of the header fields and the blob structure.

We will be using the bare-metal libfdt implementation from within uboot to parse the device tree. While the full libfdt is part of the Linux kernel, the uboot version is a stripped-down version that provides essential capabilities.

Note The ubo0t libfdt supports FDT manipulation and hence contains a lot of code that is not needed for a bare-metal read-only system. For production use, this can be stripped down to the bare minimum.

Building uboot libfdt

1. Clone the latest version of uboot from the official repository:

    git clone https://github.com/u-boot/u-boot.git
   
   

2. Change to the uboot directory and checkout the latest stable release:

    cd u-boot
    git checkout v2024.01
   

3. copy the libfdt directory to a new directory called libfdt-uboot:

    mkdir ../libfdt-uboot
    cp -r scripts/dtc/libfdt/ ../libfdt-uboot
    cd ../libfdt-uboot/libfdt
   

4. Create a makefile for the libfdt:

      touch Makefile
   

5. Add the following content to the makefile:

    include Makefile.libfdt
   
    # Name of the library to create
    LIBFDT_LIB = libfdt.a
   
    # Rule to compile each source file into an object file
    %.o: %.c
      $(CC) -c $< -o $@ -I .  # override CC with your system toolchain
   
    # Rule to create the library from the object files
    $(LIBFDT_LIB): $(LIBFDT_OBJS)
      ar rcs $@ $^
      ranlib $@
   
    # Default rule
    all: $(LIBFDT_LIB)
   
    test: $(LIBFDT_LIB)
      $(CC) -o main main.c $(LIBFDT_LIB) -I .
   
    # Clean rule
    clean:
      rm -f $(LIBFDT_OBJS) $(LIBFDT_LIB)
   

6. Write a simple C program to parse the device tree:

    #include <libfdt.h>
    #include <stdio.h>
    #include <stdlib.h>
   
    // function to read the dtb file
    static int read_dtb(char *dtbPath, void **fdt_blob) {
      FILE *fp = fopen(dtbPath, "rb");
      if (fp == NULL) {
        fprintf(stderr, "Error: Unable to open file\n");
        return 1;
      }
   
      fseek(fp, 0, SEEK_END);
      long fsize = ftell(fp);
      fseek(fp, 0, SEEK_SET);
   
      *fdt_blob = malloc(fsize);
      if (*fdt_blob == NULL) {
        fprintf(stderr, "Error: Unable to allocate memory\n");
        return 1;
      }
   
      fread(*fdt_blob, fsize, 1, fp);
      fclose(fp);
   
      return 0;
    }
   
    int main(int argc, char *argv[]) {
      const void *fdt = NULL;
      int err;
      void *fdt_blob = NULL;
   
      if (argc != 2) {
        fprintf(stderr, "Usage: %s <dtb file>\n", argv[0]);
        return 1;
      }
   
      read_dtb(argv[1], &fdt_blob);
   
      // check if the file is a valid fdt
      if (fdt_check_header(fdt_blob) != 0) {
        fprintf(stderr, "Error: Invalid device tree\n");
        return 1;
      }
   
      // get the /soc/spi tree and print the properties
      fdt = fdt_blob;
      int offset = fdt_path_offset(fdt, "/soc/spi");
      if (offset < 0) {
        fprintf(stderr, "Error: Unable to find /soc/spi\n");
        return 1;
      }
   
      int len;
      const char *prop = fdt_getprop(fdt, offset, "compatible", &len);
      if (prop == NULL) {
        fprintf(stderr, "Error: Unable to find compatible property\n");
        return 1;
      }
      printf("compatible: %s\n", prop);
   
      prop = fdt_getprop(fdt, offset, "reg", &len);
      if (prop == NULL) {
        fprintf(stderr, "Error: Unable to find reg property\n");
        return 1;
      }
   
      // print the reg address and size
      int i;
      for (i = 0; i < len / sizeof(uint32_t); i++) {
        printf("reg[%d]: 0x%x\n", i, fdt32_to_cpu(((fdt32_t *)prop)[i]));
      }
      printf("\n");
   
      // get all the spi nodes and print the properties
      int node;
      for (node = fdt_next_node(fdt, offset, NULL); node >= 0;
          node = fdt_next_node(fdt, node, NULL)) {
        char reg[32];
        const char *name = fdt_get_name(fdt, node, &len);
        if (name == NULL) {
          fprintf(stderr, "Error: Unable to find node name\n");
          return 1;
        }
        printf("node: %s\n", name);
   
        prop = fdt_getprop(fdt, node, "compatible", &len);
        if (prop == NULL) {
          fprintf(stderr, "Error: Unable to find compatible property\n");
          return 1;
        }
        printf("\tcompatible: %s\n", prop);
   
        // if compatible to "vendor,flash" print the flash-size property.
        if (strcmp(prop, "vendor,flash") == 0) {
          prop = fdt_getprop(fdt, node, "flash-size", &len);
          if (prop == NULL) {
            fprintf(stderr, "Error: Unable to find flash-size property\n");
            return 1;
          }
   
          // Assuming flash-size is a 32-bit integer
          int flash_size = fdt32_to_cpu(*(fdt32_t *)prop);
          printf("\tFlash size: 0x%x\n", flash_size);
        }
   
        prop = fdt_getprop(fdt, node, "chip-select", &len);
        if (prop == NULL) {
          fprintf(stderr, "Error: Unable to find chip-select property\n");
          return 1;
        }
        // convert property to string and print
        int cs = fdt32_to_cpu(*(fdt32_t *)prop);
        printf("\tChip Select: %d\n\n", cs);
      }
   
      free(fdt_blob);
   
      return 0;
    }
   

7. Compile and run the program:

    make test
    ./main soc.dtb
   

   The program should print the compatible and reg properties for the /soc/spi node and for each of the spi nodes.

    compatible: vendor,spi
    reg[0]: 0x800200
    reg[1]: 0x100
   
    node: flash@0
            compatible: vendor,flash
            Flash size: 0x100000
            Chip Select: 0
   
    node: temp-sensor@1
            compatible: vendor,temp-sensor
            Chip Select: 1
   
    node: accel@2
            compatible: vendor,accel
            Chip Select: 2
   

The program reads the device tree from the file soc.dtb and prints the compatible and reg properties for the /soc/spi node and each of the spi nodes.

Conclusion

Device trees are typically used in Linux systems to describe the hardware components of a system. However, device trees can be used in any bare-metal system to describe the hardware components of a system. This approach offers a great deal of interoperability and portability. It also allows for a clear separation of hardware and software, making it easier to maintain and update the software. This article showed how to use device trees in bare-metal systems.

Introduction

In this article we will go over the steps to build a bespoke Debian distro for RISC-V. We will use QEMU to emulate the RISC-V architecture and build a Debian distro for it. A bespoke OS is a custom OS that is built for a specific purpose. In this case, we will build a Debian distro that is built for RISC-V architecture. This is especially useful for embedded systems where we need full control over the OS and the packages that are installed on it.

Why Debian?

Debian is a very popular Linux distro that is used in many embedded systems. It is also the base for many other Linux distros like Ubuntu. Debian is also very popular in the RISC-V community. Debian provides a very robust package management system that allows us to install and manage packages on the OS. Debian also provides a very robust build system that allows us to build packages for the OS.

Getting Started with libe-build (lb)

Debian Live Build (lb) is a tool that allows us to build a Debian distro from scratch. It is a very powerful tool that allows us to customize the distro to our needs. It also allows us to build a distro for a specific architecture. In this article we will use lb to build a Debian distro for RISC-V architecture. Live Build uses a configuration directory to completely automate and customize all aspects of building a Live image.

Dependencies and Installation

I am using a clean debian system running on an x86 machine to build the distro.

Since we are creating a Debian distro for the RISC-V architecture on a x86 machine, we will use the qemu-user-static tool to emulate the RISC-V architecture.

Install the following dependencies:

sudo apt install qemu-user qemu-user-static qemu-system-riscv64 binutils-riscv64-linux-gnu-dbg binutils-riscv64-linux-gnu \
     libguestfs-tools binfmt-support build-essential git vim debian-archive-keyring debootstrap

In case of a non-systemd system (like WSL2), we need to enable binfmt support for qemu. This can be done by running the following command:

# running this explicitly since systemd does not start this on non-systemd systems like WSL2
sudo /usr/lib/systemd/systemd-binfmt 
# these two steps are to check if it is enabled. It should return `enabled` for the qemu-riscv64 format
sudo update-binfmts --enable qemu-riscv64
sudo cat /proc/sys/fs/binfmt_misc/status 

A side note on binfmt
binfmt lets us configure additional binary formats for executables at boot. It uses the Kernel Support for miscellaneous Binary Formats binfmt_misc that allows us to register an intrepreter to use for a specific binary format. This is useful when we want to run binaries for a different architecture on our host machine.
In our case, we are using qemu-user-static to emulate the RISC-V architecture on our x86 machine. We will use binfmt to register qemu-user-static as the interpreter for the RISC-V binary format. This will allow us to run RISC-V binaries on our x86 machine.

The magic number and other format elements used to identify the interpretor is stored in the following files
/etc/binfmt.d/*.conf
/run/binfmt.d/*.conf
/usr/lib/binfmt.d/*.conf

the systemd-binfmt command registers these formats with the kernel.
You can also register a format manually like this:
echo ':qemu-riscv64:M::\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/libexec/qemu-binfmt/riscv64-binfmt-P:OCPF' > /proc/sys/fs/binfmt_misc/register

Set up packages for risc-v with the following steps:

sudo dpkg --add-architecture riscv64
sudo apt-get install gcc-riscv64-linux-gnu g++-riscv64-linux-gnu

Then do a sudo apt update to update the package list.

We will fetch the latest version of live-build from git and build it from source.

git clone https://salsa.debian.org/live-team/live-build.git
cd live-build
mkdir install
# temp fix for missing manpages
mkdir -p manpages/fr manpages/ja
touch manpages/fr/temp manpages/ja/temp
sudo make install

Check the installation with

lb --version

Creating a distro configuration

Running lb without any arguments will create a default configuration directory in the current directory. Hpwever, we want to customize the configuration at creation time. We will use the following command to create a semi-customized configuration directory. Run this command in a directory of your choice.

export LB_BOOTSTRAP_INCLUDE="apt-transport-https gnupg"
lb config \
 --apt-indices false \
 --apt-secure false \
 --architectures riscv64 \
 --distribution unstable \
 --mirror-bootstrap http://deb.debian.org/debian/ \
 --mirror-binary http://deb.debian.org/debian/ \
 --keyring-packages "debian-archive-keyring debian-archive-keyring" \
 --security false \
 --archive-areas 'main contrib non-free' \
 --apt-options '--yes -oAPT::Get::AllowUnauthenticated=true' \
 --binary-filesystem ext4 \
 --binary-images tar \
 --bootappend-live "hostname=embeddedinn username=embeddedinn" \
 --bootstrap-qemu-arch riscv64 \
 --bootstrap-qemu-static /usr/bin/qemu-riscv64-static \
 --cache true \
 --firmware-binary false \
 --firmware-chroot false \
 --apt-source-archives false \
 --chroot-filesystem none \
 --compression gzip \
 --debootstrap-options "--keyring=/usr/share/keyrings/debian-archive-keyring.gpg --variant=minbase --include=apt-transport-https,apt-utils,gnupg,ca-certificates,ssl-cert,openssl" \
 --distribution unstable \
 --gzip-options '-9 --rsyncable' \
 --iso-publisher 'embeddedinn; https://embeddedinn.com/; vysakhpillai@embeddedinn.com' \
 --iso-volume 'embeddedinn riscv64 $(date +%Y%m%d)' \
 --linux-flavours none \
 --linux-packages none \
 --mode debian \
 --system normal \
 --updates false

Each of the options we passed are explained in the table below:

As a result of the command, the following files and directories will be created:

.
├── auto
├── config
│   ├── apt
│   ├── archives
│   ├── binary
│   ├── bootloaders
│   ├── bootstrap
│   ├── chroot
│   ├── common
│   ├── debian-installer
│   ├── hooks
│   │   ├── live
│   │   │   ├── 0010-disable-kexec-tools.hook.chroot -> /usr/share/live/build/hooks/live/0010-disable-kexec-tools.hook.chroot
│   │   │   └── 0050-disable-sysvinit-tmpfs.hook.chroot -> /usr/share/live/build/hooks/live/0050-disable-sysvinit-tmpfs.hook.chroot
│   │   └── normal
│   │       ├── 1000-create-mtab-symlink.hook.chroot -> /usr/share/live/build/hooks/normal/1000-create-mtab-symlink.hook.chroot
│   │       ├── 1010-enable-cryptsetup.hook.chroot -> /usr/share/live/build/hooks/normal/1010-enable-cryptsetup.hook.chroot
│   │       ├── 1020-create-locales-files.hook.chroot -> /usr/share/live/build/hooks/normal/1020-create-locales-files.hook.chroot
│   │       ├── 5000-update-apt-file-cache.hook.chroot -> /usr/share/live/build/hooks/normal/5000-update-apt-file-cache.hook.chroot
│   │       ├── 5010-update-apt-xapian-index.hook.chroot -> /usr/share/live/build/hooks/normal/5010-update-apt-xapian-index.hook.chroot
│   │       ├── 5020-update-glx-alternative.hook.chroot -> /usr/share/live/build/hooks/normal/5020-update-glx-alternative.hook.chroot
│   │       ├── 5030-update-plocate-database.hook.chroot -> /usr/share/live/build/hooks/normal/5030-update-plocate-database.hook.chroot
│   │       ├── 5040-update-nvidia-alternative.hook.chroot -> /usr/share/live/build/hooks/normal/5040-update-nvidia-alternative.hook.chroot
│   │       ├── 8000-remove-adjtime-configuration.hook.chroot -> /usr/share/live/build/hooks/normal/8000-remove-adjtime-configuration.hook.chroot
│   │       ├── 8010-remove-backup-files.hook.chroot -> /usr/share/live/build/hooks/normal/8010-remove-backup-files.hook.chroot
│   │       ├── 8020-remove-dbus-machine-id.hook.chroot -> /usr/share/live/build/hooks/normal/8020-remove-dbus-machine-id.hook.chroot
│   │       ├── 8030-truncate-log-files.hook.chroot -> /usr/share/live/build/hooks/normal/8030-truncate-log-files.hook.chroot
│   │       ├── 8040-remove-mdadm-configuration.hook.chroot -> /usr/share/live/build/hooks/normal/8040-remove-mdadm-configuration.hook.chroot
│   │       ├── 8050-remove-openssh-server-host-keys.hook.chroot -> /usr/share/live/build/hooks/normal/8050-remove-openssh-server-host-keys.hook.chroot
│   │       ├── 8060-remove-systemd-machine-id.hook.chroot -> /usr/share/live/build/hooks/normal/8060-remove-systemd-machine-id.hook.chroot
│   │       ├── 8070-remove-temporary-files.hook.chroot -> /usr/share/live/build/hooks/normal/8070-remove-temporary-files.hook.chroot
│   │       ├── 8080-reproducible-glibc.hook.chroot -> /usr/share/live/build/hooks/normal/8080-reproducible-glibc.hook.chroot
│   │       ├── 8090-remove-ssl-cert-snakeoil.hook.chroot -> /usr/share/live/build/hooks/normal/8090-remove-ssl-cert-snakeoil.hook.chroot
│   │       ├── 8100-remove-udev-persistent-cd-rules.hook.chroot -> /usr/share/live/build/hooks/normal/8100-remove-udev-persistent-cd-rules.hook.chroot
│   │       ├── 8110-remove-udev-persistent-net-rules.hook.chroot -> /usr/share/live/build/hooks/normal/8110-remove-udev-persistent-net-rules.hook.chroot
│   │       ├── 9000-remove-gnome-icon-cache.hook.chroot -> /usr/share/live/build/hooks/normal/9000-remove-gnome-icon-cache.hook.chroot
│   │       ├── 9010-remove-python-pyc.hook.chroot -> /usr/share/live/build/hooks/normal/9010-remove-python-pyc.hook.chroot
│   │       └── 9020-remove-man-cache.hook.chroot -> /usr/share/live/build/hooks/normal/9020-remove-man-cache.hook.chroot
│   ├── includes
│   ├── includes.binary
│   ├── includes.bootstrap
│   ├── includes.chroot_after_packages
│   ├── includes.chroot_before_packages
│   ├── includes.installer
│   ├── includes.source
│   ├── package-lists
│   ├── packages
│   ├── packages.binary
│   ├── packages.chroot
│   ├── preseed
│   ├── rootfs
│   └── source
└── local
    └── bin

25 directories, 30 files

As you would have noticed, a lot of the hooks are pointing to defaults. These hooks are used to customize the build process. We will talk about some of the hooks we would want to add and/or replace with custom hooks in order to customize our image.

Customizing the distro

There are a lot of customiations that can be done to the distro. We will just focus on a few examples here to get started.

These files are typically stored in a seperate customization folder and then copied into the autogenerated configuration directory. This allows us to keep the autogenerated configuration directory clean and easy to maintain.

All the files below are created within the config folder.

packages list

This is a relatively random list of packages that I want to install in the distro. This is not a complete list and is just for demonstration purposes.

filename: package-lists/embeddedinn-sid-server.list.chroot

mkdir -p "./config/package-lists"
cat >"./config/package-lists/embeddedinn-sid-server.list.chroot" <<"EOM"
2ping
accountsservice
apparmor
apt-utils
arping
arpwatch
at
attr
bash-completion
bc
bcache-tools
bonnie++
bridge-utils
buffer
bzip2
build-essential
ca-certificates
cifs-utils
console-data
console-setup
cpio
cron
cryptsetup-bin
curl
dnsutils
dbus
debian-ports-archive-keyring
debootstrap
dmsetup
dnsutils
dosfstools
dselect
dump
ed
eject
elinks
etherwake
ethtool
exfat-fuse
exfatprogs
f2fs-tools
file
finger
fsarchiver
ftp
fuse3
gawk
gdbserver
gdisk
gdu
genromfs
git
gpart
gpg-agent
groff-base
hdparm
hexedit
htop
iftop
inetutils-telnet
info
iotop
ipcalc
iperf3
ipmitool
iptables
iptraf-ng
iputils-ping
iputils-tracepath
irqbalance
isc-dhcp-client
isc-dhcp-common
iso-codes
isomd5sum
kbd
keyboard-configuration
keyutils
less
lftp
lldpd
lm-sensors
locales
lrzsz
lsb-release
lshw
lsof
lvm2
lz4
lzip
lzop
makedev
man-db
manpages
mbr
memtester
mime-support
minicom
mtools
mtr-tiny
ncat
netcat-openbsd
netcat-traditional
net-tools
nicstat
nmap
ntfs-3g
ntpdate
ntpsec-ntpdate
ntpsec-ntpdig
nvme-cli
nwipe
openssh-client
perl
perl-modules
openssh-server
openssl
openvpn
p7zip
partclone
parted
patch
pciutils
perl
perl-modules
policykit-1
powermgmt-base
procinfo
psmisc
python3
python3-dbus
python3-distutils
python3-gdbm
python3-gi
resolvconf
rdate
rdiff-backup
rename
rlwrap
rmlint
rsync
rsyslog
screen
sdparm
setserial
sipcalc
smbclient
socat
squashfs-tools
sshfs
ssl-cert
strace
stress-ng
stunnel4
sudo
telnet
time
tcpdump
time
tmux
tofrodos
traceroute
ucf
udftools
ufw
unp
unzip
usbutils
uuid-runtime
vim
vlan
w3m
wakeonlan
wget
whois
wipe
xorriso
xxd
xxhash
xz-utils
zerofree
EOM

user customizations

filename: includes.chroot/root/.profile

mkdir -p "./config/includes.chroot/root"
cat >"./config/includes.chroot/root/.profile" <<"EOM"
export LD_LIBRARY_PATH=$LIB:$LIBUSR:$LD_LIBRARY_PATH
EOM

filename: includes.chroot/root/.bashrc

mkdir -p "./config/includes.chroot/root"
cat >"./config/includes.chroot/root/.bashrc" <<"EOM"
export LD_LIBRARY_PATH=$LIB:$LIBUSR:$LD_LIBRARY_PATH
EOM

etc files

filename: includes.chroot/etc/resolv.conf

mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/resolv.conf" <<"EOM"
nameserver 8.8.8.8
EOM

filename: includes.chroot/etc/rc.local

mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/rc.local" <<"EOM"
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

FLAG="/var/log/firstboot.log"
if [ ! -f ${FLAG} ]
then
	echo "First boot detected" >> ${FLAG}
	echo "Performing initial setup tasks." >> ${FLAG}
	echo "Creating ssh host keys..." >> ${FLAG}
	ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -q -N ""
	ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -q -N ""
fi

ntpdate -ub 0.pool.ntp.org > /dev/null

exit 0
EOM
chmod +x "./config/includes.chroot/etc/rc.local"

filename: includes.chroot/etc/os-release

mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/os-release" <<"EOM"
NAME="embeddedinn RISC/V "
VERSION="13.0"
ID=embeddedinn
ID_LIKE=debian
PRETTY_NAME="embeddedinn RISC-V"
VERSION_ID="11.0"
HOME_URL="https://embeddedinn.com/"
SUPPORT_URL="https://embeddedinn.com/"
BUG_REPORT_URL="https://embeddedinn.com/"
EOM

filename: includes.chroot/etc/lsb-release

mkdir -p "./config/includes.chroot/etc"
cat >"./config/includes.chroot/etc/lsb-release" <<"EOM"
DISTRIB_ID=embeddedinn
DISTRIB_RELEASE=13
DISTRIB_CODENAME=Sid
DISTRIB_DESCRIPTION="embeddedinn Sid"
EOM