With the previous success of reconfiguring removable partitions via the ICAP interface, what does the AXI-lite interface allow software to do?
The aim is to investigate the facilities provided by the DFX Controller IP implemented in logic in the PL to software running in the PS. This design extends the previous blog with an AXI-Lite bus, memory mapped IP Core and software to show some of the capabilities provided for partial reconfiguration, still via ICAP.
This is part 3 of 3 in the Partial Reconfiguration series:
- Dynamic Function eXchange
- Dynamic Function eXchange with ICAP
- Dynamic Function eXchange with ICAP Driven by Software
Glossary
See the tables in both Dynamic Function eXchange and Dynamic Function eXchange with ICAP for a list of terms used here.
Introduction
It should be possible to programme the PL from PS in a bare-metal application. I note that Xilinx provide a library xilfpga on their Confluence pages. But when I try to include it in my Zynq-7000 series project it is not available. I assume this is because it is only available for Zynq UltraScale+ MPSoC and Versal platforms. So that is not going to be as easy as I thought. Also UG585 provides the instructions in the Configure the PL via PCAP Bridge Example. This looks like a sequence of address map writes and reads assuming the bitstream files are already in that address space to be read (you need to provide bitstream file source address and length), and hence a few extra complications about getting the .BIN file into the address space that I'll leave for someone else who is more familiar with the embedded software side of development.
The bridge’s DMA controller moves boot images between the FIFOs and a memory device; typically the OCM memory, the DDR memory, or one of the linearly addressable flash devices (Quad-SPI or NOR).
So instead I'll just investigate what's on offer to software for managing the DFX Controller to satisfy my remaining curiosity. It looks like DFX Controller IP Core configuration values can be read back and also set at run time. Then the removable module to use can be triggered for use. This is demonstrated by the code provided here. I have not taken any precautions to ensure that software triggers are timed separately to PL triggers from the button press, but no ill effect was observed during testing.
Design
The entire design can be recreated from a TCL script exported from the block diagram editor and augmented to include the VHDL source files required. The script is known to work in Xilinx's Vivado version 2023.2.2. The previous block diagram has been augmented to include an AXI Bus between the VHDL and the PS and hence also some AXI interconnect. The clocks and resets have been amended so they all work off the AXI interconnect and the PLL previous used by the VHDL has been removed.
DFX Controller IP core generation creates a useful file, <path>\ip\dfx_controller\documentation\configuration_information.txt, for decoding the AXI address map. Pay attention to this file because the IP core documentation is pretty awkward at best when describing the address map. It turns out that bit mapping in Dynamic Function eXchange Controller v1.0 Product Guide, PG374 does work, but I found it required some reverse engineering from the address map provided in this generated core, to the point where the documentation was unhelpful. For example, address[5:2] selects the Register within the bank, but you then have to decide how many lower bits decode the column and therefore which bits decode the row (RM number).
For bank 2 you only need address[2] to decode the column, but for bank 3 you need address[3:2] as there are 3 columns. Then the next 2 bits (in my design) decode the 4 RMs. Luckily, you can just use the AXI base address plus the offset from the text file extract below. This can be pasted into the C code as #defines.
Address Map ============================= | Virtual Socket Manager | Register | Address | +------------------------+--------------+---------+ | VS_0 | STATUS | 0X00000 | | VS_0 | CONTROL | 0X00000 | | VS_0 | SW_TRIGGER | 0X00004 | | VS_0 | TRIGGER0 | 0X00040 | | VS_0 | TRIGGER1 | 0X00044 | | VS_0 | TRIGGER2 | 0X00048 | | VS_0 | TRIGGER3 | 0X0004C | | VS_0 | RM_BS_INDEX0 | 0X00080 | | VS_0 | RM_CONTROL0 | 0X00084 | | VS_0 | RM_BS_INDEX1 | 0X00088 | | VS_0 | RM_CONTROL1 | 0X0008C | | VS_0 | RM_BS_INDEX2 | 0X00090 | | VS_0 | RM_CONTROL2 | 0X00094 | | VS_0 | RM_BS_INDEX3 | 0X00098 | | VS_0 | RM_CONTROL3 | 0X0009C | | VS_0 | BS_ID0 | 0X000C0 | | VS_0 | BS_ADDRESS0 | 0X000C4 | | VS_0 | BS_SIZE0 | 0X000C8 | | VS_0 | BS_ID1 | 0X000D0 | | VS_0 | BS_ADDRESS1 | 0X000D4 | | VS_0 | BS_SIZE1 | 0X000D8 | | VS_0 | BS_ID2 | 0X000E0 | | VS_0 | BS_ADDRESS2 | 0X000E4 | | VS_0 | BS_SIZE2 | 0X000E8 | | VS_0 | BS_ID3 | 0X000F0 | | VS_0 | BS_ADDRESS3 | 0X000F4 | | VS_0 | BS_SIZE3 | 0X000F8 |
PS Code
Banks 1, 2 and 3 are all writable as well as readable, and when the RP is shutdown it is possible to change the values. This means if you are reading a bitstream from an external source into the PL to send to the ICAP, you have the opportunity to set the bistream length correctly. One of the core's features is that there are "Up to 512 remapable software and hardware triggers per Virtual Socket". Bank 0 is variable but obvious.
I put the following code together to read the values and print them, decode the status and error bits.
#include "xil_printf.h"
#include "xparameters.h"
#define XDCFG_CTRL_OFFSET 0xF8007000
// DFX Controller Address Map
// XPAR_VHDL_CONV_I_BASEADDR + Offset below
// RO
#define DFXC_STATUS (XPAR_VHDL_CONV_I_BASEADDR + 0X00000)
// WO and is mapped to the same address as the STATUS register
#define DFXC_CONTROL (XPAR_VHDL_CONV_I_BASEADDR + 0X00000)
// RW
#define DFXC_SW_TRIGGER (XPAR_VHDL_CONV_I_BASEADDR + 0X00004)
#define DFXC_TRIGGER0 (XPAR_VHDL_CONV_I_BASEADDR + 0X00040)
#define DFXC_TRIGGER1 (XPAR_VHDL_CONV_I_BASEADDR + 0X00044)
#define DFXC_TRIGGER2 (XPAR_VHDL_CONV_I_BASEADDR + 0X00048)
#define DFXC_TRIGGER3 (XPAR_VHDL_CONV_I_BASEADDR + 0X0004C)
// Can't find which Vivado IP configuration parameter this maps to.
#define DFXC_RM_BS_INDEX0 (XPAR_VHDL_CONV_I_BASEADDR + 0X00080)
#define DFXC_RM_CONTROL0 (XPAR_VHDL_CONV_I_BASEADDR + 0X00084)
// Can't find which Vivado IP configuration parameter this maps to.
#define DFXC_RM_BS_INDEX1 (XPAR_VHDL_CONV_I_BASEADDR + 0X00088)
#define DFXC_RM_CONTROL1 (XPAR_VHDL_CONV_I_BASEADDR + 0X0008C)
// Can't find which Vivado IP configuration parameter this maps to.
#define DFXC_RM_BS_INDEX2 (XPAR_VHDL_CONV_I_BASEADDR + 0X00090)
#define DFXC_RM_CONTROL2 (XPAR_VHDL_CONV_I_BASEADDR + 0X00094)
// Can't find which Vivado IP configuration parameter this maps to.
#define DFXC_RM_BS_INDEX3 (XPAR_VHDL_CONV_I_BASEADDR + 0X00098)
#define DFXC_RM_CONTROL3 (XPAR_VHDL_CONV_I_BASEADDR + 0X0009C)
// UltraScale- only
#define DFXC_BS_ID0 (XPAR_VHDL_CONV_I_BASEADDR + 0X000C0)
#define DFXC_BS_ADDRESS0 (XPAR_VHDL_CONV_I_BASEADDR + 0X000C4)
#define DFXC_BS_SIZE0 (XPAR_VHDL_CONV_I_BASEADDR + 0X000C8)
// UltraScale- only
#define DFXC_BS_ID1 (XPAR_VHDL_CONV_I_BASEADDR + 0X000D0)
#define DFXC_BS_ADDRESS1 (XPAR_VHDL_CONV_I_BASEADDR + 0X000D4)
#define DFXC_BS_SIZE1 (XPAR_VHDL_CONV_I_BASEADDR + 0X000D8)
// UltraScale- only
#define DFXC_BS_ID2 (XPAR_VHDL_CONV_I_BASEADDR + 0X000E0)
#define DFXC_BS_ADDRESS2 (XPAR_VHDL_CONV_I_BASEADDR + 0X000E4)
#define DFXC_BS_SIZE2 (XPAR_VHDL_CONV_I_BASEADDR + 0X000E8)
// UltraScale- only
#define DFXC_BS_ID3 (XPAR_VHDL_CONV_I_BASEADDR + 0X000F0)
#define DFXC_BS_ADDRESS3 (XPAR_VHDL_CONV_I_BASEADDR + 0X000F4)
#define DFXC_BS_SIZE3 (XPAR_VHDL_CONV_I_BASEADDR + 0X000F8)
// Control Register Commands
#define DFXC_SHUTDOWN 0
#define DFXC_RESTART_WITH_NO_STATUS 1
#define DFXC_RESTART_WITH_STATUS 2
#define DFXC_PROCEED 3
#define DFXC_USER_CONTROL 4
void dfx_enable_icap();
void dfx_print_setup();
char* dfx_state_lu(u8 state);
char* dfx_error_lu(u8 error);
void dfx_print_status();
void dfx_trigger(u8 trigger);
#include <xil_io.h>
#include "dfx.h"
// https://docs.amd.com/r/en-US/ug585-zynq-7000-SoC-TRM/ICAP-Controller
// https://docs.amd.com/r/en-US/ug585-zynq-7000-SoC-TRM/Register-XDCFG_CTRL_OFFSET-Details
// XDCFG_CTRL_OFFSET @ 0xF8007000
// xsct% mwr 0xF8007000 [expr [mrd -value 0xF8007000] & 0xF7FFFFFF]
// Turn off bit XDCFG_CTRL_PCAP_PR_MASK (PCAP_PR) to enable ICAPE2 to re-program logic
void dfx_enable_icap() {
Xil_Out32(XDCFG_CTRL_OFFSET, Xil_In32(XDCFG_CTRL_OFFSET) & 0xF7FFFFFF);
}
// The RP needs to be shutdown in order to read the setup values, otherwise they come back 0x0.
void dfx_print_setup() {
u32 sizes[4];
Xil_Out32(DFXC_CONTROL, DFXC_SHUTDOWN);
sizes[0] = Xil_In32(DFXC_BS_SIZE0);
sizes[1] = Xil_In32(DFXC_BS_SIZE1);
sizes[2] = Xil_In32(DFXC_BS_SIZE2);
sizes[3] = Xil_In32(DFXC_BS_SIZE3);
xil_printf("DFXC_TRIGGER0 = RM %d\n\r", Xil_In32(DFXC_TRIGGER0));
xil_printf("DFXC_TRIGGER1 = RM %d\n\r", Xil_In32(DFXC_TRIGGER1));
xil_printf("DFXC_TRIGGER2 = RM %d\n\r", Xil_In32(DFXC_TRIGGER2));
xil_printf("DFXC_TRIGGER3 = RM %d\n\r", Xil_In32(DFXC_TRIGGER3));
xil_printf("DFXC_RM_CONTROL0 = 0x%08X\n\r", Xil_In32(DFXC_RM_CONTROL0));
xil_printf("DFXC_RM_CONTROL1 = 0x%08X\n\r", Xil_In32(DFXC_RM_CONTROL1));
xil_printf("DFXC_RM_CONTROL2 = 0x%08X\n\r", Xil_In32(DFXC_RM_CONTROL2));
xil_printf("DFXC_RM_CONTROL3 = 0x%08X\n\r", Xil_In32(DFXC_RM_CONTROL3));
xil_printf("DFXC_BS_ADDRESS0 = 0x%08X\n\r", Xil_In32(DFXC_BS_ADDRESS0));
xil_printf("DFXC_BS_SIZE0 = 0x%08X %d bytes\n\r", sizes[0], sizes[0]);
xil_printf("DFXC_BS_ADDRESS1 = 0x%08X\n\r", Xil_In32(DFXC_BS_ADDRESS1));
xil_printf("DFXC_BS_SIZE1 = 0x%08X %d bytes\n\r", sizes[1], sizes[1]);
xil_printf("DFXC_BS_ADDRESS2 = 0x%08X\n\r", Xil_In32(DFXC_BS_ADDRESS2));
xil_printf("DFXC_BS_SIZE2 = 0x%08X %d bytes\n\r", sizes[2], sizes[2]);
xil_printf("DFXC_BS_ADDRESS3 = 0x%08X\n\r", Xil_In32(DFXC_BS_ADDRESS3));
xil_printf("DFXC_BS_SIZE3 = 0x%08X %d bytes\n\r", sizes[3], sizes[3]);
Xil_Out32(DFXC_CONTROL, DFXC_RESTART_WITH_NO_STATUS);
}
void dfx_print_status() {
u32 status;
u8 state;
u8 error;
status = Xil_In32(DFXC_STATUS);
xil_printf("Status = 0x%08X\n\r", status);
state = status & 0x00000007;
xil_printf(" State : %d %s\n\r", state, dfx_state_lu(state));
error = (status >> 3) & 0x0000000F;
xil_printf(" Error : %d\n\r", error, dfx_error_lu(error));
xil_printf(" Shutdown : %d\n\r", (status >> 7) & 0x00000001);
xil_printf(" RM_ID : %d\n\r", (status >> 8) & 0x000000FF);
xil_printf("SW Trigger = 0x%08X\n\r", Xil_In32(DFXC_SW_TRIGGER));
}
char* dfx_state_lu(u8 state) {
switch (state) {
case 0:
return "Empty";
break;
case 1:
return "HW Shutdown";
break;
case 2:
return "SW Shutdown";
break;
case 3:
return "Clearing BS";
break;
case 4:
return "Loading";
break;
case 5:
return "SW Startup";
break;
case 6:
return "Reset RM";
break;
case 7:
return "Loaded";
break;
default:
return "";
}
}
char* dfx_error_lu(u8 error) {
switch (error) {
case 0:
return "No Error";
break;
case 1:
// The fetch path was asked to load a 0 byte bitstream
return "Bad Configuration";
break;
case 2:
// The ICAP returned an error code while loading the bitstream
return "BS Error";
break;
case 3:
// Access to the ICAPE3 was removed during a bitstream transfer. This error is only possible when the device to be managed is an UltraScale or UltraScale+ device.
return "Lost Error";
break;
case 4:
// There was an error fetching the bitstream from the configuration library.
return "Fetch Error";
break;
case 5:
// The ICAP returned an error code while loading the bitstream and there was an error fetching the bitstream from the configuration library.
return "BS & Fetch errors";
break;
case 6:
// Access to the ICAPE3 was removed during a bitstream transfer, and there was an error fetching the bitstream from the configuration library. This error is only possible when the device to be managed is an UltraScale or UltraScale+ device.
return "Lost & Fetch errors";
break;
case 7:
// A compressed bitstream ended at an invalid place in the decompression algorithm.
return "Bad Size Error";
break;
case 8:
// A compressed bitstream was received in the incorrect format.
return "Bad Format Error";
break;
case 15:
// An unknown error occurred.
return "Unknown Error";
break;
default:
return "Unassigned error";
}
}
void dfx_trigger(u8 trigger) {
Xil_Out32(DFXC_SW_TRIGGER, trigger);
}
Demonstration
The code below calls dfx_print_status() twice per loop so that I could check the effect of competing with the PS software by pressing the button on the board to perform a PL trigger of the ICAP reconfiguration.
#include "sleep.h"
#include "xil_printf.h"
#include "xparameters.h"
#include "dfx.h"
int main() {
u32 rm = 3;
xil_printf("Start Running\n\r");
xil_printf("-------------\n\r");
dfx_print_setup();
dfx_enable_icap();
while (1) {
usleep(500000); // us
dfx_print_status();
// Software requests a sepcific RM to be loaded by the DFX Controller logic over AXI-Lite
// Counting backwards means its can't be a request from digital logic
dfx_trigger(rm);
rm = (rm-1) % 4;
usleep(500000); // us
dfx_print_status();
}
}
I can capture the output of the PS software using a serial port monitor such as PuTTY. In my case the device to monitor is on COM4, but your PC's "Device Manager" will tell you which COM port to use, but only if your FPGA card is turned on. Below is the captured data, showing the memory offsets and sizes of the RMs configured in the DFX Controller IP. It also shows the RM loaded by software couting backwards (the opposite direction to the PL trigger).
Start Running ------------- DFXC_TRIGGER0 = RM 0 DFXC_TRIGGER1 = RM 1 DFXC_TRIGGER2 = RM 2 DFXC_TRIGGER3 = RM 3 DFXC_RM_CONTROL0 = 0x00000019 DFXC_RM_CONTROL1 = 0x00000019 DFXC_RM_CONTROL2 = 0x00000019 DFXC_RM_CONTROL3 = 0x00000019 DFXC_BS_ADDRESS0 = 0x00000000 DFXC_BS_SIZE0 = 0x00000DD4 3540 bytes DFXC_BS_ADDRESS1 = 0x00001000 DFXC_BS_SIZE1 = 0x00000DC8 3528 bytes DFXC_BS_ADDRESS2 = 0x00002000 DFXC_BS_SIZE2 = 0x00000ED8 3800 bytes DFXC_BS_ADDRESS3 = 0x00003000 DFXC_BS_SIZE3 = 0x00000DEC 3564 bytes Status = 0x00000007 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 0 SW Trigger = 0x00000000 Status = 0x00000307 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 3 SW Trigger = 0x00000003 Status = 0x00000307 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 3 SW Trigger = 0x00000003 Status = 0x00000207 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 2 SW Trigger = 0x00000002 Status = 0x00000207 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 2 SW Trigger = 0x00000002 Status = 0x00000107 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 1 SW Trigger = 0x00000001 Status = 0x00000107 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 1 SW Trigger = 0x00000001 Status = 0x00000007 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 0 SW Trigger = 0x00000000 Status = 0x00000007 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 0 SW Trigger = 0x00000000 Status = 0x00000307 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 3 SW Trigger = 0x00000003 Status = 0x00000307 State : 7 Loaded Error : 0 Shutdown : 0 RM_ID : 3 SW Trigger = 0x00000003
Conclusions
You are unlikely to be loading RMs from internal PL memory via the ICAP simply because a useful design will have larger RMs and hence larger space requirements, and the ability to load bitstreams from SPI flash or DDR via PCAP achieves the same ends without consuming logic resources. It is more likely that you will read the RMs into the FPGA through Select I/O pins and require digital logic to perform operations on that stream to prepare the bitstream for the ICAP interface. However the bitstream interface is AXI-MM not AXI-S, so another piece of IP will be required for the stream to memory map conversion, e.g. Data Mover IP core. Software might need to be used to configure the start address and length of the processed bitstream coming from the Data Mover IP to the DFX Controller IP. (NB. Internal BlockRAM was only used in this design as a simplifcation to check the functionality of the DFX Controller IP.)