VEGAS: VErsatile GBT Astronomical Spectrometer
VEGAS is a next generation instrument that will expand the capabilities of the current spectrometer at the Green Bank Telescope, to address the increased bandwidth and spectral resolution needed with the new 7-pixel Ku-band focal plane array receiver. The NRAO website gives a detailed description of the instrument and it's scientific objectives.
Team
Marty Bloss, Patrick Brandt, Hong Chen, Jayanth Chennamangalam, Paul Demorest, Gregory Desvignes, Terry Filiba, John Ford, David Frayer, Bob Garwood, Glenn Jones, Joe Masters, Randy McCullough, Guifré Molera, Karen O'Neil, Aaron Parsons, Jason Ray, Anish Roshi, Simon Scott, Amy Shelton, Mark Wagner, Galen Watts, Dan Werthimer
Block Diagram
Software
ROACH Firmware
- BORPH - A modified Linux kernel that runs on the PPC, and allows for interaction with shared memory on the FPGA.
- KATCP - Allows for remote command and control of the firmware, over the network.
- Roach Monitor - Monitors voltages and temperatures of ROACH boards.
HPC Software
The HPC (which is a cluster of CPUs and GPUs) will be running a modified version of the backend software for GUPPI (guppi_daq). There are two different versions of the HPC software:
- vegas_hpc_hbw : the HPC software for high-bandwidth modes (the CPU merely accumulates)
- vegas_hpc_lbw : the HPC software for low-bandwidth modes (the GPU implements a PFB)
Depending on the desired mode, one of the two HPC applications must be run on the HPC. Note that a separate instance of the HPC application must be started on each CPU/GPU node in the cluster.
Starting and Stopping the HPC Software
To start the HPC software, simply run an instance of vegas_hpc_* application on each CPU. The application does not require any command-line parameters. The one optional parameter that can be passed to the process is "--disk", which causes the application to write SDFITS files for debugging purposes.
To stop the HPC software, simply kill the process on each CPU node.
Outputting Data
By default, the HPC software does not write to disk. The HPC software writes integrated spectra to a shared memory buffer. The M&C process (external to the HPC software) then reads the spectra from the shared memory buffer and writes the spectra to disk.
However, if the "--disk" flag is passed to the HPC software, via the command line, the HPC software will create a disk thread which writes the spectra from the output buffer to SDFITS files on disk. This mode is merely for testing purposes, and will not be used during normal operation.
Details on the shared memory buffers can be found in the Memo on the Shared Memory Buffers and Memo on the Critical Settings for the HPC, both of which can be found in the section on Technical Documentation on the VEGAS HPC Software Architecture below.
Installing the HPC Software
Installation notes for the VEGAS HPC software can be found here [Installation notes]
Technical Documentation on the VEGAS HPC Software Architecture
- Memo on the Critical Settings for the HPC: vegas_hpc_critical_settings.pdf
- VEGAS HPC Software Developer Documentation: vegas_hpc_dev_doc.pdf
- VEGAS HPC GPU thread code-level documentation: VEGAS GPU Code Documentation
Older documents that have been subsumed by the HPC Developer Documentation:
- Memo on Control and Timing for the ROACHs and HPC: gbt_memo_control_timing.pdf
- Memo on the Shared Memory Buffers vegas_memo_shmem.pdf
- SPEAD Packet Format for High-Bandwidth Modes: spead_format_high_bandwidth.pdf
- SPEAD Packet Format for Low-Bandwidth Modes: spead_format_low_bandwidth.pdf
Network Packets
[SPEAD] protocol - The SPEAD protocol allows for a very robust and flexible means for streaming packets. This protocol is used for packets that are sent from the FPGA to the HPC. Detailed information of the packet format at [VEGAS Packet Format]
Hardware
- 9x ROACH boards
- 18x ADC1x3000-8 Single-8 bit, 3000Msps National ADC083000 ADC
- 9x CPUs
- 9x GPUs
- 9x [Myricom 10GbE NICs]
Firmware
The firmware is the code that runs on the FPGA. It is available from the VEGAS project [git repository] (deprecated: [gbt_devel git repository]). We are using the 11.5 MSSGE toolflow for all modes.
Monitor and Control of the ROACH Boards
Vegas FPGA Register Definitions
All shared memory on the FPGA will be accessed over the 100Mbit port using the corr and katcp python packages that communicate with the tcpborphserver running on the PPC. Monitoring of ROACH voltages and temperatures will be done by the Actel Fusion and can be accessed via the Xport (see ROACH Monitor for details).
