USBpix - USB based readout system for ATLAS FE-I3 and FE-I4

Introduction

The USB based FE-I3 readout system was developed as an alternative to the common 'TurboDAQ' (TPLL, TPCC) system for a couple of applications. See the presentation for further details and specifications. Since the system has a modular structure, it will be adapted to serve also as a FE-I4 r/o system - requiring a new module adapter card and changes in FPGA firmware and DAQ application.

USBpix system for FE-I3 single chip read-out.

A CERN e-group mailing list - atlas-pixel-usbpix - has been created for communication concerning the USBpix system and its software; e-mail access is restricted to group members. In order to (un-)subscribe, please use the e-group Web-interface.

Hardware

The USBpix hardware is based on a multi-purpose FPGA card (Multi-IO board) and dedicated adapter- and support cards for FE-I3 and FE-I4 single chip setups.

Multi-IO board

• User manual (rev. 1.03)
• schematic and layout prints (rev. 1.03)

FE-I3 setup

• adapter card: schematic and layout prints
• single chip card: schematics and layout of the new single chip board, Gerber files
  • Note: All single chips boards which are delivered with the USBpix systems are configured so that the 3.3 V for the LVDS chips are supplied via the USB system. If you like to test these chips with a TPLL/TPCC system you can either feed the 3.3V via the TPCC with an extra power supply or you set the solder jumper SJ8 instead of SJ7 to feed the voltage directly via the lemo-connector on the single chip board.
  • SCB_known_issues.pdf: Known issues of single chip board.

FE-I4 setup

• adapter card
  • Rev. 1.0a (first two prototypes only): schematic and layout prints
  • Rev. 1.1: schematic and layout prints
  • Rev. 1.1a: schematic and layout prints
• single chip card
  • Rev. 1.0: schematic and layout prints, Gerber files
  • Rev. 1.1: schematic and layout prints, pitfalls and changes to rev. 1.1

Miscellaneous

• FE-I4 emulator card schematics
• Connectors for the flat ribbon cable between the adaper card and single chip card (see: http://www.kel.jp/english/product/Half_8900/8925e.pdf)
  • KEL 8925-50-179-F (connector w/o ejector) or
  • KEL 8925E-50-179-F (connector with ejector and strain relief) or
  • KEL 8925R-50-179-F (connector with ejector and w/o strain relief)
  • cable: 50 connectors 0,25" round conductor flat cable (for example 3M 3749/50)
• Schematics and layout of the Quad Module Adapter Card
• Presentation on test system development (Pixel Upgrade Meeting, CERN 27.06.2008)
• USB interface card documentation

Software

The application software for the USBpix system is based on the ATLAS PixLib package. This collection of C++ classes has originally been developed to provide access to the ATLAS Pixel detector RODs. The hardware specific layer of this package has been adapted to access the FE pixel chips via the USB/FPGA card. The user interface to configure and operate the system is an adapted version of STcontrol which uses ROOT and Qt to implement a graphical user interface.

More information can be found in the STcontrolUserGuide (for FE-I3) or STcontrolI4 (for FE-I4).

PixLib software related documentation

• https://twiki.cern.ch/twiki/bin/view/Atlas/STControlDocPage (original, i.e. ROD-related STcontrol documentation); for the USB-related version of STcontrol, see the STcontrolUserGuide (for FE-I3) or STcontrolI4 (for FE-I4).

• PixLib documentation: https://twiki.cern.ch/twiki/bin/view/Atlas/PixLib
• How to do scans with PixLib: PixLibScans, containing an updated version of the ROD-related information on https://twiki.cern.ch/twiki/bin/view/Atlas/PixLibScans
• General PixDAQ developer ressources: https://savannah.cern.ch/projects/atlaspixel/
• DBEditor documentation (view and modify scan data and config. files): https://twiki.cern.ch/twiki/bin/view/Atlas/DBeditorTool
• SctPixelRod documentation: http://sctpixel.home.cern.ch/sctpixel/SctPixelRodSoftware.html
• Module analysis documentation: ModuleAnalysis
• Slides from EVO-tutorial on USBPix in April 2011 (based on I4 release 1.2 and I3 release 1.5): on indico

Operating systems / build environments supported

• Windows XP / Windows 7, 32-bit, Microsoft Visual Studio (Express) 2008
• Linux SLC5, 32-bit, gcc 4.3 (NB: not really fully supported, but the I4 code is tested occasionally and seems to work as of release 1.3)
• Other Linux distributions: Read USBPixReleaseNotes for USBPixReleaseNotesI4 for more information. Please share your experience.

A new driver is currently under development! The driver is based on libusb (for Linux/Unix) and WinUSB? /libusb-windows-backend (Windows XP/Vista/7). This development is currently in beta state. More information can be found in the Using libusb section.

USBpix software

For running STcontrol on Windows XP, compiled binary files (executables and DLLs) can be used which will be found under \host\trunk\bin (or \host\tags\x.y\bin for revisioned versions). In addition to the main application STcontrol other executables are available:

• USBpixTest/ USBpixI4Test, a small test application which performs FE-I3/FE-I4 configuration and simple scans
• SiUSBMan, control applet for the USB device interface (for configuration and firmware updates)

There are two different subversion repositories (svn) available. One for FE-I3 (also referred to as USBpix) and one for FE-I4 (USBpixI4? ). These repositories can be found at:

• FE-I3: http://icwiki.physik.uni-bonn.de/svn/USBPix/
• FE-I4: http://icwiki.physik.uni-bonn.de/svn/USBpixI4/

The directory structure is as follows:

• \device --> source code used on the device side (FPGA, microcontroller)
• \doc --> additional documentation (see links above for basic documentation on the USBpix components)
• \host --> PixLib? based software for the host side (user interface, libraries, driver, microcontroller and FPGA configuration files, etc.)
• \pcb --> project files for the module adapter PCBs (Altium Designer project files, Gerber production files)

Installation

Windows XP / Windows 7

There are three ways to get a working USBpix test system depending whether you want to use precompiled binaries or compile the code by yourself:

Beginner Mode: Intended for use by the end-user who don't want to spend much time on installing the USBpix test system.

1. Download the most recent setup file from USBpix svn:

• USBpix 1.5 (FE-I3)
• USBpixI4 2.0.5 (wo USB Man) (FE-I4)

1. Install the package by double-clicking on the setup file.
2. Select all components. NOTE: The setup for USBpix 1.5 requires ROOT 5.26/00 and USBpix 2.0 requires ROOT 5.32.00. The USBpixI4? setup will automatically install or update to the required version of ROOT. This needs internet connection. An offline installer for USBpixI4? can be found here.
3. Select the destination folder (e.g. c:\USBpix)
4. Program shortcuts were added to USBpix folder in the Windows Start Menu but not on the desktop.
5. NOTE: Do not forget to update the microcontroller firmware.

Medium Mode: Intended for use by the experienced user who wants to use the stable release but wants to compile his own binaries (and eventually using different versions of required software mentioned under Additional Prerequisites)

1. Checkout the USBpix svn (URLs above)
2. Follow the instructions under Additional Prerequisites: Install Microsoft Visual Studio 2008, Qt and ROOT, extract Qwt.
3. Open Command Prompt and change to your Qwt directory (e.g. c:\qwt-5.2.1)
4. Run "qmake qwt.pro"
5. Run "nmake"
6. Open Command Prompt and change the directory "\USBPix\host\tags\release-[x].[y]\" (FE-I3) or "\USBpixI4\host\tags\[x].[y]\" (FE-I4)
7. Run "setup.bat"
8. Run "nmake clean"
9. Run "nmake"
10. The applications and shared libraries can be found in "%DAQ_BASE%\bin", which is included in your PATH variable and thus can directly be called from the Commant Prompt. For the execution from Windows Explorer or using a shortcut, make sure you have "%QTDIR%\bin;%ROOTSYS%\bin;%QWTDIR%\lib" added to your PATH variable.
11. Install/Update the MultiIOBoard? driver which can be found under "\USBPix\host\tags\release-[x].[y]\driver\winXP\dpinst.exe"
12. NOTE: Do not forget to update the microcontroller firmware.

Insane Mode: Intended for use by the professional user who wants to use the most recent development version from trunk. Do not expect any mercy. Update regularly and read the svn log.

1. Checkout the USBpix svn (URLs above)
2. Follow the instructions under Additional Prerequisites: Install Microsoft Visual Studio 2008, Qt and ROOT, extract Qwt.
3. Open Qt Command Prompt (or set environment variables accordingly) and change to your Qwt directory (e.g. c:\qwt-5.2.1)
4. Run "qmake qwt.pro"
5. Run "nmake"
6. Open Command Prompt and change the directory to "\USBPix\host\trunk\" (FE-I3) or "\USBpixI4\host\trunk\" (FE-I4)
7. Run "setup.bat"
8. Run "nmake clean"
9. Run "nmake"
10. The applications and shared libraries can be found in "%DAQ_BASE%\bin", which is included in your PATH variable and thus can directly be called from the Commant Prompt. For the execution from Windows Explorer or using a shortcut, make sure you have "%QTDIR%\bin;%ROOTSYS%\bin;%QWTDIR%\lib" added to your PATH variable.
11. Install/Update the MultiIOBoard? driver which can be found under "\USBPix\host\trunk\driver\winXP\dpinst.exe"
12. NOTE: Do not forget to update the microcontroller firmware.

Linux

NOTE: No support provided, so use at your own risk!

For USB driver installation on a Linux system follow the instructions as described in usbpix_install.pdf or in section Using libusb.

It is recommended to build the applications from the source tree:

• needed environment variables: ROOTSYS, QTDIR; if you want to use USBPixTest, set also QWTDIR; see section Additional Prerequisites (USBPixTest is urrently not compiling under linux, so no need to install QWT)
• go to the base directory, i.e. "\trunk" or "\tags\x.y"
• source setup_noTDAQ.sh
  • on SLC4, in case afs is enabled and running, you can also source setup.sh instead which will then build the application including ROD-related code (only for old DSP code, so obsolete)
  • both setup scripts only exist for bash shells and similar; if you are using (t)csh, please modifiy scripts accordingly
  • in case you want to use GPIB capabilities, execute the following after having sourced either of the setup scripts:
    • export GPIB_FLAG=-DHAVE_GPIB
    • export PIX_GPIB=$DAQ_BASE/PixGPIB

• make genlib pixlib

After make finished, the applications and shared libraries can be found in $DAQ_BASE/bin, which is included in your PATH and LD_LIBRARY_PATH variables, i.e. the applications will run simply by typing their name in the command line.

This means in turn that you have to run the setup(_noTDAQ).sh script every time before executing any of the application.

Updating the microcontroller firmware

A new release often contains a new version of the microcontroller firmware. Note: Do not forget to update the microcontroller firmware. Flashing the microcontroller is currently supported only under Microsoft Windows.

Perform the following steps:

1. Connect the USBpix-board, LED6 should turn green.
2. Open "SiUSBman.exe" in the bin-folder or use the Start Menu entry. The MultiIOBoard? should appear in the list.
3. In the "Options" menu select "Expert mode".
4. Select the "USB firmware" tab in the lower part of the window.
5. Click on the file browser button ("...") next to the "Flash EEPROM" button and browse to "USBPix/host/tags/release-[x.y]/config" and select the file "usbpix.hex".
6. Press the "Flash EEPROM" button. IMPORTANT: Be patient! Do not touch the system while performing the update!
7. Press the switch "S1" on the MultiIOBoard? to restart the microcontroller.
8. The microcontroller will now boot up with the new firmware. The MultiIOBoard? should appear in the list showing the new firmware version.

Additional Prerequisites

• Compiler:
  • Windows: Microsoft Visual Studio 2008 (VC++ 9). If you don't have a Microsoft Visual Studio 2008 license available, you can also use the free Microsoft Visual Studio Express 2008. If you only want to use the precompiled binaries from svn, Microsoft Visual C++ 2008 SP1 Redistributable Package should be enough. Do not use Visual Studio 2010 (VC++ 10).
  • Linux: Use the gcc compiler and (g)make that comes with your distribution.
• Nokia Qt 4.6.3 LGPL:
  • Windows: Qt libraries, precompiled (194 MB)
  • Linux: Qt libraries, source code (161 MB)
• Qwt 5.2.1:
  • Windows / Linux: Qwt libraries, source code (2.5 MB)
• ROOT 5.26/00:
  • Windows: ROOT libraries, precompiled (52.5 MB)
  • Linux: ROOT libraries, source code (27.3 MB)
• Environment Variables:
  • ROOTSYS: Base directory of the root installation (e.g. "c:\root").
  • QTDIR: Base directory of the Qt installation (e.g. "c:\Qt\4.6.3").
  • QWTDIR: Base for the Qwt installation (e.g. "c:\qwt-5.2.1").
  • VCDIR (Windows / VC++): Base directory of the Microsoft Visual C++ installation (e.g. c:\Program Files\Microsoft Visual Studio 9.0\VC).
  • Windows: In case you want to use the exe-files in \USBPix\host\[trunk\tag]\bin without running any setup-script, make sure you have "%QTDIR%\bin;%ROOTSYS%\bin;%QWTDIR%\lib;" added to your PATH variable. Note that you need exactly the same versions of Qt, Qwt and ROOT as specified above in order to use those exe-files directly.

Using libusb

Usage of USBpix with libusb is currently in beta state and only included in the FE-I4 sources (Insane Mode) so far!

To comile the USBpix Software using libusb follow these steps from a clean :

• Set the Environment Variable USBPIX_LIBUSB to USE_LIBUSB .
• Run the setup script (setup_noTDAQ.sh or setup.bat)
• Build the Software with make / nmake

Addtionally the libusb driver has to be installed. The Windows driver can be found at "USBpixI4/host/trunk/drivers/WinLibUSB". To install the driver just run dpinst_x86.exe or dpinst_x64.exe depending on your system.

To use libusb under Linux you need to load libusb via your package manager. For Ubuntu 11.04 this can be done via "apt-get install libusb-1.0-0 libusb-1.0-0-dev".

To run the USBpix Software with a non root user on a Linux system you have to adjust the udev rules. Using Ubuntu 11.04 this can be done via creating a new file named "/etc/udev/rules.d/50-usbpix.rules" with the following content:

SUBSYSTEM!="usb", GOTO="usbpix_rules_end" SYSFS{idVendor}=="5312", SYSFS{idProduct}=="0200", GROUP="USBPIX", MODE="0660" SYSFS{idVendor}=="5312", SYSFS{idProduct}=="0201", GROUP="USBPIX", MODE="0660" SYSFS{idVendor}=="5312", SYSFS{idProduct}=="0300", GROUP="USBPIX", MODE="0660" SYSFS{idVendor}=="0547", SYSFS{idProduct}=="2131", GROUP="USBPIX", MODE="0660" SYSFS{idVendor}=="0547", SYSFS{idProduct}=="0080", GROUP="USBPIX", MODE="0660" SYSFS{idVendor}=="04b4", SYSFS{idProduct}=="8613", GROUP="USBPIX", MODE="0660" LABEL="usbpix_rules_end"

Afterwards execute the following commands, where (USERNAME) has to be replaced by the name of your user:

sudo groupadd USBPIX sudo usermod -a -G USBPIX (USERNAME)

To make sure that all rules are applied restart your system.

By default the libusb based libraray has many debug outputs to see where problems occur. This also slows down the system. To avoid these outputs you can change SF_DEBUG_USB in libsiusb/interface_libusb.h in line 28 to 0 and recompile the library.

Release Notes

The release notes for each tagged version of the USBpix software can be found on the page USBPixReleaseNotes for the FE-I3 version or USBPixReleaseNotesI4 for the FE-I4 version.

Known Issues

libthread.dll - unresolved export function

Symptom: The libthread.dll is part of the ROOT installation and is called by STControl. However if you have also installed the ISE FPGA Software from Xilinx, a DLL with the same name is found on the system too. Usually ROOT and Xilinx installations add their directories to the PATH variable. So if the Xilinx directory is found in the PATH variable before the ROOT directory, the wrong DLL is called (and vice versa if you start the Xilinx ISE).

Workaround: Start STControl within a DOS box which sets a PATH variable without the Xilinx directory.

Current and temperature measurements

Symptom: Due to a bug on the adapter card the supply current and the temperature measurements are not working (supply voltage read back is working though).

Workaround: With cutting some traces and adding a few wires the bug can be fixed. If current and temperature measurement capabilities should urgently be needed, we can provide instructions to fix the PCB. However a new card with a corrected layout is currently being produced and will be made available to the users if needed.

Using USB hubs

Connecting the board via a USB hub and not directly to a USB socket is known to cause problems at least when running fedora linux. If you observe problems (e.g. error messages from TUSBDevice), try to connect directly to a socket.

FE-I3 and MCC Documentation

Information on the FE-I3 chip and module controller chip MCC 2.1 can be downloaded here:

• ATLAS FE-I3 documentation (V3.0)
• FE-I3 NIM conference paper
• MCC-I2.1 Specifications: command decoder
• MCC-I2.1 Specifications: event builder
• MCC-I2.1 Specifications: output format
• MCC-I2.1 Specifications: receiver
• MCC-I2.1 Specifications: register bank
• MCC NIM conference paper

Calibration data for the FE-I3 single chips which have been delivered together with the USBpix systems can be downloaded here:

• ATLAS FE-I3 single calibration data

General reading on the pixel detector:

• The ATLAS Pixel electronics and sensor paper 2008 JINST 3 P07007
• Habilitation thesis on the ATLAS Pixel Detector of Jörn Grosse-Knetter (contains details about scanning with FE-I3)
• Documentation of production tests with MCC/FE-I3 modules: ATL-IP-QP-0144 (you will need EDMS access to read this)

Inventory

Tables of USBpix compontents can be found here.

FAQ (FE-I4 operation and tests with USBpix)

..can be found here.

USBpix EUDAQ Integration

More information can be found here