Technology Blogs

We play with technology. Sometimes we discover things we think are worth sharing.

Verification of Clock Domain Crossing Topologies

Posted by on 25 May 2022 in FPGA, VHDL, TCL

Even if you are aware of all the rules about how to mitigate metastability from passing signals from one clock domain to another, i.e. clock domain crossings, it is all too easy to miss a clock domain crossing (CDC) by inspection using Mark 1 Eye Ball, or to forget some of the constraints, or to foul up the cell name in XDC, or to overlook some finer aspect of risk mitigation because the delivery of the design is divided amongst a team, who may be working to different standards. Vivado includes a TCL function report_cdc to aid the designer, and it has proven effective in locating issues that need addressing. Here are some examples of the finer points that the function identifies.

Schematic Markup

Verification of Clock Domain Crossing Topologies
Standard two flop-flip synchroniser (aka. double retime).

This figure illustrates how the standard schematics from Vivado are marked up by a TCL procedure to display flops coloured by clock domain and ASYNC_REG registers marked by a red diamond.

Example Reports

Vivado User Interface

Verification of Clock Domain Crossing Topologies
Example 'report_cdc' output to a window in the GUI.

Vivado Console Output

report_cdc -details
INFO: [Timing 38-91] UpdateTimingParams: No interconnect No Cell Dly, Speed grade: -1, Delay Type: min_max.
INFO: [Timing 38-191] Multithreading enabled for timing update using a maximum of 6 CPUs
INFO: [Timing 38-433] Consider using Xilinx recommended XPM_CDC modules to avoid Critical severities
INFO: [Timing 38-314] The report_cdc command only analyzes and reports clock domain crossing paths where clocks have been defined on both source and destination sides. Ports with no input delay constraint are skipped. Please run check_timing to verify there are no missing clock definitions in your design, nor any unconstrained input port.
Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.
--------------------------------------------------------------------------------------
| Tool Version : Vivado v.2019.1.1 (win64) Build 2580384 Sat Jun 29 08:12:21 MDT 2019
| Date         : Mon May 23 22:52:18 2022
| Host         : Rievaulx running 64-bit major release  (build 9200)
| Command      : report_cdc -details
| Design       : report_cdc
| Device       : 7k70t-fbv676
| Speed File   : -1  PRODUCTION 1.12 2017-02-17
--------------------------------------------------------------------------------------

CDC Report

ID      Severity  Count  Description
------  --------  -----  --------------------------------------------------
CDC-1   Critical      5  1-bit unknown CDC circuitry
CDC-2   Warning       1  1-bit synchronized with missing ASYNC_REG property
CDC-3   Info          3  1-bit synchronized with ASYNC_REG property
CDC-4   Critical      1  Multi-bit unknown CDC circuitry
CDC-10  Critical      1  Combinational logic detected before a synchronizer
CDC-11  Critical      2  Fan-out from launch flop to destination clock
CDC-15  Warning       2  Clock enable controlled CDC structure detected

Source Clock: clk_src
Destination Clock: clk_dest
CDC Type: No Common Primary Clock

Row  ID      Severity  Description                                         Depth  Exception   Source (From)                                   Destination (To)
---  ------  --------  --------------------------------------------------  -----  ----------  ----------------------------------------------  -------------------------------------------------
  1  CDC-1   Critical  1-bit unknown CDC circuitry                             0  False Path  cdc_invalid_data_slow_fast_i/dv_i_reg/C         cdc_invalid_data_slow_fast_i/data_out_reg[0]/CE
  2  CDC-1   Critical  1-bit unknown CDC circuitry                             0  False Path  cdc_invalid_data_slow_fast_i/dv_i_reg/C         cdc_invalid_data_slow_fast_i/data_out_reg[1]/CE
  3  CDC-4   Critical  Multi-bit unknown CDC circuitry                         0  False Path  cdc_invalid_data_slow_fast_i/data_i_reg[1:0]/C  cdc_invalid_data_slow_fast_i/data_out_reg[1:0]/D
  4  CDC-1   Critical  1-bit unknown CDC circuitry                             0  False Path  cdc_invalid_data_slow_fast_i/dv_i_reg/C         cdc_invalid_data_slow_fast_i/data_valid_out_reg/D
  5  CDC-15  Warning   Clock enable controlled CDC structure detected          0  False Path  cdc_validated_data_slow_fast_i/data_i_reg[0]/C  cdc_validated_data_slow_fast_i/data_out_reg[0]/D
  6  CDC-15  Warning   Clock enable controlled CDC structure detected          0  False Path  cdc_validated_data_slow_fast_i/data_i_reg[1]/C  cdc_validated_data_slow_fast_i/data_out_reg[1]/D
  7  CDC-3   Info      1-bit synchronized with ASYNC_REG property              2  False Path  cdc_validated_data_slow_fast_i/dv_i_reg/C       cdc_validated_data_slow_fast_i/dv_reg[0]/D
  8  CDC-3   Info      1-bit synchronized with ASYNC_REG property              2  False Path  reg_capture_a_reg/C                             sync_reg_1/reg_retime_reg[0]/D
  9  CDC-2   Warning   1-bit synchronized with missing ASYNC_REG property      2  False Path  reg_capture_c_reg/C                             sync_reg_3/reg_retime_reg[0]/D
 10  CDC-11  Critical  Fan-out from launch flop to destination clock           2  False Path  reg_capture_d_reg/C                             sync_reg_4/reg_retime_reg[0]/D
 11  CDC-11  Critical  Fan-out from launch flop to destination clock           2  False Path  reg_capture_d_reg/C                             sync_reg_5/reg_retime_reg[0]/D
 12  CDC-10  Critical  Combinational logic detected before a synchronizer      2  False Path  reg_capture_e_reg/C                             sync_reg_6/reg_retime_reg[0]/D
 13  CDC-1   Critical  1-bit unknown CDC circuitry                             0  False Path  reg_capture_b_reg/C                             sync_reg_bad_i/flags_out_reg[0]/D
 14  CDC-1   Critical  1-bit unknown CDC circuitry                             0  False Path  reg_capture_e_reg/C                             sync_reg_bad_i/reg_retime_reg/D
 15  CDC-3   Info      1-bit synchronized with ASYNC_REG property              2  False Path  xpm_cdc_single_inst/src_ff_reg/C                xpm_cdc_single_inst/syncstages_ff_reg[0]/D
Example report_cdc output to the console.

Notice the former has classifications of "Safe", "Unknown" and "Unsafe" not given in the console output.

Xilinx "QuickTake" Video

Xilinx have their own instructional video on this topic called "Using Report CDC to Analyze CDC Structural Issues". This provides an overview of the command we'll cover. Later I provide some hand crafted code to purposefully illustrate some of these errors and test report_cdc's ability to identify them.

Using Report CDC to Analyze CDC Structural Issues.

Synchroniser Errors

The following examples show the most commonly found errors in real designs. It is not exhaustive over all topologies report_cdc is supposed to discern, just the ones you are most likely to encounter.

Single Bit Synchronisers

Verification of Clock Domain Crossing Topologies
Identifying insufficient synchronisation registers.

The second synchronising register is absent altogether. A second register needs adding to the topology with the ASYNC_REG property (VHDL attribute). See Single Bit Synchroniser.

CDC-1   Critical  1-bit unknown CDC circuitry
report_cdc result.
Verification of Clock Domain Crossing Topologies
Missing ASYNC_REG property on the second synchronising register.

The multiple stage synchroniser has been identified correctly and the absent ASYNC_REG called out.

CDC-2   Warning   1-bit synchronized with missing ASYNC_REG property
report_cdc result.
Verification of Clock Domain Crossing Topologies
Logic before the synchronising registers prolongs the settling time for sampling and increases chances of metastability.

There should be no combinatorial logic between the launch register in the source clock domain and the first registers in the destination clock domain. See Combinatorial Logic.

CDC-10  Critical  Combinational logic detected before a synchronizer
report_cdc result.
Verification of Clock Domain Crossing Topologies
The path between two ASYNC_REG registers should be short and hence without logic.

Perhaps goes without saying, but there should be no logic between ASYNC_REG registers, since that obstructs what Vivado is supposed to achieve with registers marked with ASYNC_REG attributes or properties. report_cdc will identify this case as a single-stage synchroniser error.

CDC-1   Critical  1-bit unknown CDC circuitry
report_cdc result.

Multi-Bit Synchronisers

Verification of Clock Domain Crossing Topologies
Multi-bit CDCs should be considered arrays of independent bits not a data bus needing to remain in synchronisation across the width.

Multi-bits crossing are warned because perhaps its a data bus which should use a FIFO? In this case there was no attempt at synchronisation at all. See Multi-Bit Synchroniser.

CDC-4   Critical  Multi-bit unknown CDC circuitry
report_cdc result.
Verification of Clock Domain Crossing Topologies
The same signal should not be synchronised multiple times as each output may change on adjacent clock cycles. Any derived products from these signals are out of synchronisation.

The fanout warning is specific to multiple re-timings of the same control signals. As with a multi-bit CDC, there's no synchronisation guarantee across all copies, so any derived products must be assumed to be out of synchronisation. This is fine if their paths never meet again. The suggestion is to synchronise once and re-use multiple times. This situation can easily occur in teams building a design, and each person does what they need without having access to the larger picture to decide to factor out the common logic. See Fanout.

CDC-11  Critical  Fan-out from launch flop to destination clock
report_cdc result.
Verification of Clock Domain Crossing Topologies
The fanout itself is not a problem, just the multiple synchronisations of the same control bit.

Just to confirm that not all fanout is bad, this example with a fanout of 2 does not raise an report warning as its not a repeat synchroniser, or as it probably detects, not repeating the same clock domain crossing.

Verification of Clock Domain Crossing Topologies
The logic driving the clock enable (CE) pin must be sourced from the same clock domain as the clock of the register.

The clock enable register pin must be driven from logic in the same clock domain as the registers clock pin. This is a warning as the dynamic timing of when to enable the CE pin must be verified in simulation to ensure the D input has settled before being enabled, or is otherwise managed. This case occurs when passing data qualified by a "data valid" signal across clock domains as only the data valid signal needs to be synchronised whilst the data is allowed to settle before sampling in the new clock domain. See CE Controlled CDC.

CDC-15  Warning   Clock enable controlled CDC structure detected
report_cdc result.

Xilinx Parameterized Macros (XPM)

Verification of Clock Domain Crossing Topologies
All the constraints are provided for you with the XPM_CDC_SINGLE component from Xilinx. It also provides run-time checks for dynamic timing checks.

For UltraScale and more recent devices you can of course use the XPM library components for synchronisation. These provide the topologies and the constraints required for successful CDC. In addition they provide Verilog simulation assertions that look for dynamic timing issues that static timing tools cannot find. In the case of XPM_CDC_SINGLE, each signal to cross clock domains needs to be held stable on input to the synchroniser two samplings by the destination clock domain. In practice that means three destination clock cycles as that's easy to code in Verilog assertions. See XPM_CDC_SINGLE.

Verification of Clock Domain Crossing Topologies
XPM_CDC_SINGLE provides Verilog assertions to check the dynamic timing of the input to ensure that it is stable for at least 2 samplings by the destination clock, approximated to 3 destination clock cycles.

Note you need a Verilog simulation license to take advantage of these checks, VHDL only simulators will not execute the checks. The simulation checks are controlled via the SIM_ASSERT_CHK generic value.

# ** Warning: [XPM_CDC_ARRAY_SINGLE S-1] Input data (src_in[0]) at 599970000000 is not stable long enough to be sampled twice by the destination clock. Data in source domain may not transfer to destination clock domain.
#    Time: 599970 ns Started: 599954 ns  Scope: <path>.<register>[x] File: C:/Xilinx/Vivado/2020.1/data/ip/xpm/xpm_cdc/hdl/xpm_cdc.sv Line: 1044
XPM library component warning for synchroniser inputs changing too soon.

XPM FIFO Warnings

Xilinx IP cores use XPM FIFOs internally, but do not (always?) provide waivers for what we ought to assume are verified usages with CDC. This means report_cdc will list many lines of warnings for components that perhaps should not have them.

Verification of Clock Domain Crossing Topologies
Xilinx IP cores use XPM FIFOs internally, but do not provide waivers for what we ought to assume are verified usages with CDC.
CDC-15  Warning   Clock enable controlled CDC structure detected
report_cdc result.

Managing Report Volume

The report_cdc can return 100s of line items. Many are the same error over a 32-bit data bus so can be knocked off quickly once the first bit has been checked, but that still leaves a lot of checks to be done. The best way to not lose you place in the manual verification is to waiver those you have checked to be safe, and fix those you discover are a warning for a reason (obviously).

Waivers

Here's an example of a waiver that been used to silence warnings in Xilinx's XPM FIFOs. The assumption here is that Xilinx know what they are doing and can be trusted to get things right, therefore they don't need further checking.

create_waiver -type CDC -id CDC-15 \
    -from [get_pins {<instance_name>/inst/gen_async_conv.axisc_async_clock_converter_0/xpm_fifo_async_inst/gnuram_async_fifo.xpm_fifo_base_inst/gen_sdpram.xpm_memory_base_inst/*/RAM*/CLK}] \
    -to [get_pins {{<instance_name>/inst/gen_async_conv.axisc_async_clock_converter_0/xpm_fifo_async_inst/gnuram_async_fifo.xpm_fifo_base_inst/gen_sdpram.xpm_memory_base_inst/gen_rd_b.doutb_reg_reg[*]/D}] \
    -user <the team> \
    -description {Xilinx XPM} \
    -tags {Verified}
Waiver specification for the XPM FIFO used inside many Xilinx IP Cores, but left un-waivered for report_cdc to find.

Recommendations

The table layout in the GUI is a little misleading at first glance. There is no such thing as a "safe critical". Each row of the summary table is for a pair of clocks. The severity is the worst case for that clock pair. "Critical" severity means at least one issue that is either "unsafe" or "unknown". So look at the "Category" column for a round up of how bad things are in the design. The text report to the console avoids this ambiguity.

SeverityAction
CriticalThese are most likely to be a real issue that needs fixing.
WarningsThese are a mixture that needs sorting through. Either they are a CDC issue that needs a fix or waiver them by putting the create_waiver command in a SCOPED_TO_REF XDC file. Too often they are simply signals attempting to cross a clock domain without any attempt at a proper treatment.
Actions to take on report_cdc results.

Conclusions

It might be tempting to throw in a double retime and a false path and think you are done. We probably knew about the ASYNC_REG property or attribute. If we're honest, we've probably overlooked several details such as no combinatorial logic between the last register in the source clock domain and the first register in the destination clock domain. We've probably not considered the fanout issue, mainly because we've not noticed. And we might be surprised that an input signal to a synchroniser violated the multiple destination clock-cycle hold time because it escaped "Mark 1 Eye Ball" in a long simulation, so steps need to be taken to decide how to stabilise or hold that signal constant long enough. report_cdc seems to do a good job of highlighting issues. All the 'criticals' have been right so far, and many of the warnings did identify a standard CDC problem. Make report_cdc part of your standard build verification and include waivers in constraints files to silence warnings that truly can be ignored.

References

  1. Github Source Code
  2. Visualising Clock Domain Crossings in Vivado
  3. UG912 explains the ASYNC_REG property
  4. Xilinx UG906, Understanding the Clock Domain Crossings Report Rules
  5. Xilinx UG906, Summary by Clock Pair

No feedback yet


Form is loading...