Static Proton and Heavy Ion Testing of the Xilinx Virtex-5 Device

This paper presents proton and heavy ion static results for the latest Xilinx field-programmable gate arrays (FPGAs). The paper analyzes static bit cross-sections, resources, multiple-bit upsets (MBUs) and angular effects.


I. INTRODUCTION
Static random access memory (SRAM) based fieldprogrammable gate arrays (FPGAs) have been the focus of much radiation research in recent years [1]. SRAM-based FP-GAs realize circuits through a combination of programmable routing and look up tables (LUTs), both of which use SRAM to support reconfiguration. The reconfigurable nature makes them an interesting possibility for space-based applications, as reconfiguration allows the designers to mitigate radiationinduced upsets at the application level and alter the circuit while deployed. The fine-grained parallelism available from these devices make them well-suited for implementing the signal processing applications commonly used in space-based processing. Unfortunately, the SRAM cells used to hold the programming data is soft and SEUs on these devices affect the state of the circuit, the LUTs and the programmable routing resources. This paper will present results from proton and heavy ion testing of the latest commercial device, the 65-nm Virtex-5 (XC5VLX50), from July to October of 2006, regarding bit cross-section, multiple-bit upsets (MBUs), resources, and angular effects. Fig. 1 shows a diagram of a Virtex-5 device. This device is laid out in columns of resources. The left, right and middle columns are devoted to the input/output blocks (IOBs) that connect the reconfigurable fabric to the device's pads. The interior region of the device is predominantly configurable logic blocks (CLBs) where the circuit functionality is realized. Occasional columns of clocking, BlockRAM (BRAM), Block-RAM interconnect (BRAMi) and digital signal processing (DSP) resources are interspersed between the CLB columns. The XC5VLX50 has nearly 13 million SRAM bits to program the device with 8.5 million bits in the CLB resources and two million in BRAM resources. The remaining 2.5 million bits are in the IOBs, BRAMi, and the DSPs.
While FPGAs are very regular in their physical layout, each resource is unique in layout and radiation characteristics. Manuscript  Any skews in the data cause by angular rotation of the test fixture in the beam gives us insight into the physical spacing in the layout. Likewise the distribution of MBUs in each resource gives us insight into how to space a circuit through out the device to minimize the effect of MBUs. As the Virtex-5 family comes in many flavors with different sizes and resource distributions, these resource characteristics can be used to project overall device characteristics for untested parts. This paper covers the most common resources: CLBs, BRAMs, BRAMi, IOBs, and DSPs. The remainder of this paper covers our test methodology (II), and our test results (III).

II. TEST METHODOLOGY
In this section we present our hardware/software test fixture, analysis software, and experimental methodology. We tested the commercial version of the XC5VLX50 Virtex-5 device on bulk silicon, as the military/aerospace version on an epitaxial layer is not available yet. Our previous research has found that the commercial and the military/aerospace devices are very similar in radiation characteristics. Los Alamos National Laboratory (LANL), NASA Goddard Space Flight Center (GSFC), and the Xilinx Radiation Testing Consortium (XRTC) collected heavy ion data at the K500 Cyclotron at Texas A&M University (TAMU), and the 88" Cyclotron facility at Lawrence Berkeley National Laboratory (LBNL). Table I lists the linear energy transfer (LET), total fluence, species and angle for each facility. LANL and NASA's GSFC collected the proton data at Indiana University Cyclotron Facility (IUCF) for 200 and 65 MeV. We were able to collect data at several angles, which are listed in Table II. A single part was used at IUCF and it received a total ionizing dose of 320 KRads.

A. Test Fixture
Our test fixture is a combination of commercial hardware and custom software. The hardware test fixture, shown in Fig. 2, is two Xilinx AFX series development boards (one Virtex-II and one Virtex-5) biased nominally. The Virtex-II board communicates to the host computer through a USB card and controls the Virtex-5 board during irradiation. Custom software performs a readback of the programming data (or bitstream), differences the reference bitstream and the readback data, and immediately reconfigures the device with the reference bitstream. The FPGA is completely reprogrammed and the differential bitstream saved to hard drive every second in this scheme, which allows us to test at high fluences without accumulating upsets. We can collect approximately 3,600 differential readbacks per hour. Minimal statistics are taken in real time while irradiating the part. Data analysis is reserved as a post-collection process, and is described in Section II-B.
Part of our interest in the radiation characteristics lies in the MBU response on the device, since we are interested in the role of MBUs in triple-modular redundancy (TMR) defeats in FPGAs. Therefore, while we test at high fluences, we are mindful that distinguishing MBUs is dependent on keeping the fluence/readback low enough to not create MBUs out of coincident single bit upsets (SBUs). Coincident SBUs are physically adjacent SBUs created from two or more ionized particles and are indistinguishable from MBUs caused by a single ionized particle. Our experimental methodology attempts to maximize data collection speed and minimize the probability of coincident SBUs so that MBUs can be disambiguated. For this work, we collected between 15-300 upsets per readback and the worst case probability of coincident SBUs in this test data is 0.006%. Therefore, the MBU statistics are not coincidental. A discussion of coincident SBUs, including the method for estimating the probability of coincident SBUs in the data set, can be found in our previous work [2]. Despite a low probability of coincident SBUs, we were able to collect between 7,000-22,000 upsets per hour in proton testing, and 108,000-2,800,000 upsets per hour in heavy ion testing.
As well as MBUs, we are interested in the angular effects of these devices. Unlike SRAM devices, these devices are laid out heterogeneously with each resource having its own layout and the programmable routing network breaking the layout into a grid. Therefore, we expect a response that is more complex than the one made by simply increasing the LET. To study these effects, we rotated the device in two different directions, as shown in Fig. 3, to change the beam's angle of incidence. Since the device is columnar in nature, we tested the response when the columns were upright (θ = 0) and when the columns were on their side (θ = 90). Next the beam's angle of incidence (φ) was changed by slanting the device relative to the beam. Several different φ's were tested for both θ's, to get an idea of how the angular effects changed the radiation characteristics.

B. Post-Data Collection Analysis
Our post-data collection software that we created focuses three basic aspects: pre-analysis data cleansing; classification of upsets by event, resource and shape; and statistical analysis. Our data collection process attempts to record all events that occur on the device as long as the event does not cause the test fixture to fail. Therefore, a number of single-event functional interrupts (SEFIs) are collected. As many of the SEFIs cause a large percentage, if not all, of the device to be upset at one time and are often reported separately to the SEU data, we removed these records from the data set so they would not skew results. This pre-analysis data cleansing step is done using a statistical technique called jackknifing and was tuned to make certain the SEFIs are being removed from the data set but that good data is not removed. Since SEFIs have much smaller cross-sections than the programming data's bit cross-section by several orders of magnitudes, the jackknifing removes only a small percentage of the entire data set. Once the SEFIs are removed, the data is processed. The first step is to cluster upsets into SBUs and MBUs. Two upsets are considered adjacent if the upsets lie within each other's eight neighboring memory cells, as shown in Fig.4. Adjacent upsets are classified as MBUs. Each readback data sample is clustered independently to maintain temporal locality. Next, the affected resource for each upset is determined. Finally, the shape of the MBU is determined. Unlike the earlier devices in the Virtex family, the Virtex-5 has demonstrated a wide array of MBU shapes and are reported in terms of the number of affected rows and columns (i.e., row x col), called a bounding box. An example of a (2x2) bounding box for a 3-bit MBU is shown in Fig. 5. Bounding boxes are used to determine the compactness of MBUs.
Once all of the upsets are classified by event, resource, and shape, the analysis software creates several different types of histograms. In this paper, we will present distributions based on event size, resource, and a combination of the two. The analysis software also does a number of histograms and distributions to determine whether the device is uniformly upset by the number of events, the number of SBUs, and the number of MBUs based on the entire device, as well as by resource. Finally, a number of basic statistics on the data set are determined, such as the minimum number of upsets per readback, maximum number of upsets per readback, average number of upsets per readback, expected number of rows affected, expected number of columns affected, expected compactness, and the entire data set's skew and kurtosis.

III. PROTON AND HEAVY ION RESULTS
For this research we are interested in bit cross-sections as well as resource and orientation skews. While we collected many normal incidence data points, we collected angular data as well, which led to some surprising discoveries. First, we present the normal incidence data.

A. Bit Cross-Sections
We first analyzed the data for bit cross-sections. We are concerned with both upset and event bit cross-sections, where upset bit cross-sections are calculated from the raw number of upsets in the data set (i.e., each MBU counts as the number of bits upset) and the event bit cross-sections are calculated from the number of single ionized particle-induced events in the data set (i.e., each MBU counts as one event). The proton event bit cross-section for the 5VLX50 is 8.61x10 −14 ± 4.90x10 −16 cm 2 /bit for 200 MeV and 6.37x10 −14 ± 1.17x10 −15 cm 2 /bit for 65 MeV. The upset and event bit cross-sections for heavy ions can be found in Fig. 6. The event bit cross-section for the highest tested LET of 68.3 MeV-cm 2 /mg is 5.73x10 −8 cm 2 /bit. In our previous research on the Virtex-I, Virtex-II and Virtex-4 devices [2], we found that the event bit crosssection for MBUs was larger than the SEFI cross-section. In the Virtex-5 device, the MBU event bit cross-section is not only larger than the SEFI cross-section, but is larger than the SBU bit cross-section at the high LETs. Therefore, under some circumstances MBUs are expected to be the predominant response.
A comparison between the Virtex-5 device and it's most immediate Virtex predecessor, the Virtex-4, is not simple. While we did test another similarly sized commercial Virtex-4 device (XC4VSX35), the distribution of resources makes device-to-device comparisons unwise, although resource-toresource comparisons can be made. Comparing the event bit cross-sections by resource type for these two devices is made even more difficult because the two families have different sub-saturation and saturation characteristics. At sub-saturation LETs the CLB event bit cross-section is 3.5 times smaller in the Virtex-5 than it is the Virtex-4, whereas the BRAM event bit cross-section is 3 times larger in the Virtex-5 than the Virtex-4. This behavior switches in the saturation regions, though. In the saturation region the CLB event bit crosssection is 4 times higher in the Virtex-5 than it is the Virtex-4, whereas the BRAM event bit cross-section is 3.5 times smaller in the Virtex-5 than the Virtex-4. When these results are used to estimate what a similarly sized Virtex-4 with the XC5VLX50's resource distribution, the estimated Virtex-4 event bit cross-section is slightly smaller in the sub-saturation region, but is about a factor of two larger in the saturation region.  Looking at the distribution of events by size allows us to get an idea of range of MBU sizes that could be invoked on orbit. Above 38.05 MeV-cm 2 /mg more than half of all events are MBUs. Fig. 9 and 8 show the distribution of event sizes for for proton-induced events and heavy-ion-induced events, respectively. These graphs show that as LET increases in heavy ion irradiation or Energy increases in proton irradiation, the percentage of 1-bit events decreases and the percentage of 2to 4-bit events increase gradually. At the high LETs in heavy ion not only are 2-bit events roughly equivalent in frequency as 1-bit events, but 3-and 4-bit events together represent 25% of the total number of events. In proton testing, the nuclear recoil reaction does not provide as strong of an MBU response than the heavy ion direct ionization reaction and most MBU events are limited to 2-bit events. While the proton MBU response is limited, the Virtex-5 device is the first Virtex device that has any proton-induced MBUs larger than 4-bits and even had very occasional 9-bit MBUs. The average event size for 200-MeV proton-induced upsets is 1.13 and for heavy-ioninduced upsets at the highest tested LET of 68.3 MeV-cm 2 /mg is 2.57. Most of the 200-MeV proton MBUs are 2-bit (1x2) events (50%), 2-bit (2x1) events (32%), and 3-bit (2x2) events (8%). The average distribution of heavy-ion-induced MBU events over all LETs is: 56% 2-bit (2x1), 21% 2-bit (1x2), 7% 2-bit (2x2), 5% 3-bit (3x1), 3% 4-bit (2x2), and 3% 4-bit (3x2). These results indicate a strong correlation between row adjacencies and MBUs.

C. Resource Analysis
We have also determined the distribution of events by resource, since protecting resources from radiation-induced upsets differs by resource. While the BRAM events dominate the very low LETs, CLB resources dominate above 3.03 MeV-cm 2 /mg. The first bar on Fig. 10 ("area") indicates the percentage of device area each resource takes. Unlike the rest of the Virtex devices, the heavy ion distribution of events by resource approaches the amount of space each resource has on the device. In the Virtex-I and Virtex-II, the resource distributions had an unusually large response from the IOB and an unusually low response from the BRAM. In the Virtex-4, Fig. 10. Heavy Ion Percentage of Events by Resource the resource distribution had an unusually high response from the BRAM and an unusually low response from the CLBs. In the Virtex-5, with 200-MeV protons, the distribution of events is as such: 42% CLB, 46% BRAM, 4% DSP, 5% IOB, and 4% BRAMi. There is a slight shift (3%) of the CLB events to BRAM events in 65-MeV protons, otherwise the distribution of events by resource is the same as 200-MeV protons. In the Virtex-5, as with the Virtex-4, the BRAM resources tend to upset with more column adjacencies than row adjacencies, whereas the opposite is true of CLBs.
While most resources had very similar event distributions for the first time in the Virtex line, almost all of the MBUs in heavy ion and proton involving five or more bits were in the CLB resources. We are concerned about MBUs in the CLB region since even 2-bit MBUs can compromise redundancy schemes for mitigation in the earlier generation devices. Of the three predominate sub-resources for CLBs (routing network, lookup tables (LUTs), and state space) on average 70% of MBUs in the CLB region occur in the routing network, 20% in the LUTs, and 10% in LUT state space that determines the mode of the LUTs and user flip flops (i.e., whether the LUT is a LUT, RAM or ROM or a flip flop is reset low or high), which also follows the percentage of area devoted to these subresources. The high percentage of MBUs in the CLB routing network is a concern, since our research into TMR defeats While MBUs in the LUT region are most likely to manifest as observable output errors, redundancy mitigation schemes are able to mask these errors as long as the MBU occurs entirely within one LUT. Therefore, the concern for redundancy-based mitigation schemes is that MBUs in the CLB region do not affect affect multiple LUTs or span from a single LUT into

D. Angular Effects
Finally, we did extensive angular testing on the Virtex-5 part, which lead to several interesting discoveries. In general, both the heavy ion event bit cross-section (Fig. 11) and the heavy ion and proton MBU percentage increased with angle (Figs 12 and 13). The proton bit cross-sections were all within a factor of two of the normal incidence bit-cross-sections. We found that the overall difference was larger with lower LET species (Neon, Boron, Argon) than with higher LET species (Krypton, Xenon). Similarly, 65-MeV protons had a greater change as φ increased than with 200-MeV protons and have nearly the same percentage of MBUs as with 200-MeV protons. We also found that the event bit cross-sections and MBU percentages were inconsistent with normal incident results for similar LETs. Not only is there a higher probability of an MBU, but the MBUs shift to larger sizes (13% 5-and 6bit events in Krypton at a 60-degree angle). With both θ's we found that a 10% shift in the event distribution from BRAM events to CLB events. There were also differences in the θ rotations, as shown in Fig. 14 for Argon. We found differences in the distribution of events by size. With θ = 0, as φ increased, BRAM MBUs increased in percentage and size. The larger BRAM MBUs were often larger in column adjacencies. With θ = 90, as φ increased, CLB MBUs increased in percentage and size. The larger CLB MBUs were often larger in row adjacencies. These results indicated that the two largest resources, CLB and BRAM, are laid out differently such that the resources have different responses with respect to the θ rotation. In 200 MeV proton there is only a slight change (1%) between the θ rotations where φ ≤ 60, but there was a noticeable difference when the device was irradiated at φ = 90 (device edge facing the beam). When θ = 90, the percentage of MBUs and the event bit-cross-section continues to increase, whereas both quantities decrease when θ = 0. This result indicates that the active region is wider in the column direction than in the row direction.
While angular testing is useful on it's own merit because the space environment is omni-directional, these results call into question the usefulness of rotating the test fixture in the beam to increase the LET. Given the non-homogeneous layout of these devices, the angular effect does not highlight an increased LET but the heterogeneity of the device's layout. While increasing the LET should increase the percentage of MBUs, increase the percentage of large MBUs, and cause a shift in the distribution of events by resource, the results do not change by expected amounts. The event distribution for angular data has a greater diversity and a larger contribution from MBUs than a similar LET at a normal incidence, as shown in Figs 15 and 16. In this case the LET has increased 7% from 68.3 MeV-cm 2 /mg to 72.8 MeV-cm 2 /mg, but the increase in the percentage of MBUs (59% to 72%) indicates a much higher change in LET. In the normal incidence data only 1% of the event distribution includes 5-bit and larger events, whereas 16% of the event distribution for the angular data includes 5-bit and larger MBUs. Also, the shift in the distribution by resource seems to highlight the eccentricities in the resource layout. Therefore, we feel that mixing normal incidence and angular data should be avoided for reporting purposes.

IV. CONCLUSIONS
In this paper we have presented results for proton and heavyion static testing for the Xilinx Virtex-5 device. The largest event bit cross-section is 5.73x10 −8 cm 2 /bit in heavy ions and 8.61x10 −14 cm 2 /bit in 200-MeV protons. There has been an increase in MBUs from previous generations with most events involving 1-to 4-bits at a normal incidence. This device also exhibits strong angular effects with an increase in bit crosssections and percentage of MBUs in heavy ions. Further study of this device will indicate whether these results will have an impact on redundancy-based fault mitigation schemes.
V. ACKNOWLEDGMENTS