Overview
This article reviews the architecture and implementation details for the Private Island ® Machine Learning Framework. This framework enables data collection, custom pre-processing, and real-time inferencing of network data. Inferencing may be integrated within the machine learning module, a daughter board, and/or remote via a LAN port to perform inferencing (deep learning) on a local or remote model server. Pre-processing includes packing receive data & variables and creation of well-formed data tensors for direct input to an inferencing engine.
This description is current as of Git commit 8363d90c, which is currently being used on a beta verion of the Betsy™ FPGA maker board.
This article uses the following conventions:
- module names are written with italics
- FSM state names are CAPITALIZED.
- variable and wire names are lower case and bold
Definitions / Terms:
- Block: A collection of DUs per frame
- Data Unit (DU): data that is clocked into the MLE from a receive module and processed as a unit of data.
- Field: byte or group of bytes within the DU that have a specific meaning (e.g., fp32, int16, or 32-bit IPv4 address).
- Frame: Time frame while the MLE event is active.
- Machine Learning Engine (MLE): The Verilog module being defined in this article.
As the system block diagram below depicts, the ml_engine is an autonomous, independent block. For GigE applications, the interface & module are nominally clocked at 125 MHz.
UDP is utilized for the transmission of MLE generated packets in order minimize overhead and simplify the overall architecture. Note that UDP is an appropriate protocol since the need to replay an MLE generated packet is highly unlikely. For the case where a remote model server is utilized, a translator is typically utilized (e.g, UDP -> TritonClient via the PI-EXP application)
Note that the figure above is specific to Betsy and utilizes RGMII. Other implementations exist that utilize SGMII and XGMII for higher data rates (e.g., 2.5G and 10G).
The figure below shows the data flow and I/O into and out of the MLE.
Engine Architecture
The ml_engine asserts an enable for each configured receive module one at time using a handshake. Each write into the engine's memory is counted and used to determine the overall packet size when initiating its transmission into the soft switch. At the end of each DU write sequence, a flag is set to indicate end of write (DU).
Events are considered active during the period when data transfers occur. An event ends when there are no more receiving modules with data to write into the MLE.
The ml_engine module can be instantiated more than once and dedicated to specific PHYs or other data receive modules.
The MLE architecture relies on one clock. Handshakes with RX modules may require CDC sequencers to prevent metastable conditions.
As shown in the figure below, the engine utilizes a pair of DPRAMs as multiple buffered memory. While one DPRAM is emulating a FIFO to receive data from an RX module, the other DPRAM is being used to write data into the switch for transmit to an inferencing resource that could be either internal or external (e.g., Ethernet PHY, daughter board connector, remote model server, etc.).
If complex internal processing is not required, the second (dpram_s1) DPRAM can be bypassed using the DIRECT_OUTPUT directive.
In general, the MLE implementation must take care not to read from the same DPRAM address while it is being written. This is a straightforward goal to accomplish using FSMs.
The enable logic controls the assertion of each enable signal for a particular RX module. The empty input for each unused module interface should be tied high to disable the interface.
Receive modules do not write the DU size since the data transfer may end abruptly due to the data received or other events external to the ml_engine.
DPRAMs are not cleared between events, so the implementation must account for this.
FSM_0 controls the assertion of evt_start and evt_active. Active is negated after there is no more data to be transferred (all empty lines are asserted).
FSM_2 follows FSM_1, but the start time is skewed to prevent DPRAM collisions. A counter (not shown) controls the amount of skew.
pkt_sz is a count register that is incremented on each write into the second DPRAM to determine transmit packet size.
The ml_engine interface supports prioritization of the data that is transferred into the soft switch.
The module may be configured to use internal resources, external, or a combination of both for tensor formation.
The internal processing block can assert action & trigger signals as needed to other modules and does not necessarily need to be synchronized to other events inside the MLE module.
Having multiple blocks of data in dpram_s1 enables buffering of the blocks until the switch can transfer the data. It is an open issue whether the processing should stall or data should be overrun if a DPRAM is full. Each block is read out from dpram_s1 as a continuous block of data.
Engine Configuration
The engine can be configured to initiate a transmit depending on either time interval or a particular trigger.
A DIRECT_OUTPUT Verilog directive is currently supported to simplify the architecture and eliminate / bypass the second DPRAM. In this case, only the FSM_0 and FSM_3 state machines are active.
Event Processing
The figure below shows the event time slicing between the three FSMs. Each delay is set individually via parameters.
MLE Input DU Definition
The fields of each data unit (DU) are defined below. Note that the size field is written by the ml_engine itself based on its internal counter at the end of each message. One module can send multiple data blocks by negating empty after one clock.
Depending on the code read during Step 1, all, some, or none of the data may be processed in the DU. The same is true as each field is read and potentially transferred to the Processing and Inferencing blocks.
- SRC ID
- Code / Type
- Data (last byte, set MSB as end of DU flag)
- Size or Checksum
Note that the size field conveys the number of bytes and includes both data and header / footer fields.
MLE Packet Output Definition
The MLE will automatically and periodically transmit a packet to an inferencing resource, which may be located internally, on a daughter board, or remotely via a LAN interface. The format of the packet is described below.
The MLE header contains a 16-bit time step / sequence number based on a 32-bit internal FPGA counter (lower bits conveyed). For Betsy / GigE Ethernet, each bit represents 8 ns, and the counter will roll over approximately every 34 seconds.
RX Module Interface
The figure below shows a block diagram of the MLE interface within an RX module (e.g., mac_rgmii). The interface is designed with the assumption of Cross Domain Clocking (CDC) between the rx_clk and mle_clk. Note that the synchronizers also introduce a deterministic amount of delay between events to ensure the RX data is stable and locked before it is written into the MLE.
The negation of empty triggers the MLE data transfer event, and enable starts the actual data transfer sequence. It's critical that data written into the MLE isn't changing during the event.
mle_d and mle_d_o may vary in implementation from a single register, multiple registers, and / or DPRAM.
mle_state is used to sequence the write data from the RX module into the MLE and relies on an mle_cnt register for its implementation.
Examples
To be added...
