A FRAMEWORK FOR COLLABORATIVE REVIEW OF CANDIDATE EVENTS IN HIGH DATA RATE STREAMS: THE V-FASTR EXPERIMENT AS A CASE STUDY

Astropulse is part of a series of sky surveys for radio pulses being conducted by the Search for Extraterrestrial Intelligence (SETI) at the University of Berkeley (Siemion et al. 2010). The Astropulse project conducts a survey of the sky from the Arecibo Observatory in Puerto Rico, searching for short (microsecond) broadband radio frequency pulses. While Astropulseʼs use of Areciboʼs enormous single dish telescope affords excellent sensitivity, V-FASTRʼs ability to perform continent-scale baseline interferometery yields much greater positional accuracy when attempting to localize the source of a signal.

As a variant of the SETI@home project, Astropulse utilizes the same distributed, collaborative volunteer computing infrastructure accumulated over the years by that effort to perform a number of computationally intense transformations and calculations of the data in an attempt to better classify the origin of any signals detected. The use of volunteer computing to perform units of computational work is an appealing approach that obviates the need to directly acquire sufficient hardware for the processing demands. However, the fully automated nature of the approach is not a natural fit for V-FASTRʼs manual review requirement.

GalaxyZoo ⁴ is an Internet-based project that relies on the help of volunteers to classify a very large database of galaxy images recorded by either the Sloan Digital Sky Survey or the Hubble telescope. Users are asked to classify galaxies based on shape, color and direction of rotation, and report on possible unidentified features. The rationale behind human intervention is that manual classification is more accurate and insightful than any algorithm that can currently by undertaken by an automatic program. To date, the project has met with success that far exceeded expectations: more than 250,000 volunteers have helped classify millions of images, resulting in the confirmation of important scientific hypothesis, the formulation of new ones, and the discovery of new interesting objects.

While Galaxy Zooʼs tactic of appealing to volunteers to mitigate the challenge of image classification at scale is attractive, the paradigm does not translate well to the V-FASTR setting due to differences in the nature of the archives between the two projects. Whereas Galaxy Zoo permits its volunteer reviewers to leisurely peruse and mine a largely static image archive, the rapidly growing data volumes associated with ongoing V-FASTR observations dictate that reviews must be regularly scheduled to keep the project within its resource limits.

Foldit (Cooper et al. 2010) is a collaborative online protein folding game developed by the Center for Game Science at the University of Washington, and it represents a "crowd-sourced" attempt to solve the computationally challenging task of predicting protein structure. Proteins, chains of amino acids, play a key role in a wide range of human diseases, but comparatively little is known about how they contort themselves into the specific shapes that determine their function. Because of the scale and complexity of the challenge, the researchers behind Foldit have turned to the puzzle-solving capabilities of human beings for assistance. After learning the rules on simple challenges, players compete against one another to design alternative protein structures, with the goal of arriving at an arrangement that minimizes the total energy needed to maintain the shape.

Foldit has created an environment in which the unknown and diverse strategies of its human participants become a core strength. Furthermore, by presenting the scientific activity as a competitive game, the project, which currently boasts over 400,000 players, has shown that it is possible to recruit and leverage human processing power at scale. This provides an interesting model for other projects, including V-FASTR, which at some point may rely upon a human element to augment or improve automated processes.

As described in Section 2, our group has developed substantial experience in the design and implementation of data systems for a broad range of scientific domains. In each case, a close working relationship with members of the project science team was an essential ingredient to the success of the project, and our experience developing an online candidate review framework for V-FASTR was no different. As software engineers familiar with the challenges inherent in scientific data management, our intuitions about the technical challenges of the system served us well in scoping out the project timeline. However, it was our early and regular communication with members of the V-FASTR science team that was critical to obtaining the domain knowledge necessary to make accurate assumptions, and in the early identification of issues. The current system architecture, covering both the back and front end elements, is a direct result of an ongoing feedback loop between the science and software teams.

As mentioned in Section 2, V-FASTR is a commensal experiment that scans for fast transients in data that is already being collected as part of the regular third-party use of the VLBA instrument. As such, the experiment maintains a "guest" status on the NRAO computing infrastructure. Consequently, care must consistently be taken not to overtax NRAO system resources, including disk storage, CPU time, and network bandwidth. These physical constraints motivated many of the architectural decisions described in the following sections.

Each V-FASTR data product may contain hundreds of files, rooted at a top-level job directory, and includes two types of products: filterbank data (up to $\sim 100$ GB per job) and baseband voltage data (up to $\sim 10$ GB per job). The total data storage capacity available to V-FASTR is just $\sim 8$ TB, enough to contain $\sim 800$ jobs of $\sim 10$ GB each (on average). Because products are produced at a average rate of $\sim 10$–20 per day (but sometimes in the hundreds), the storage would be exhausted within a few weeks unless products are periodically reviewed by the science team analysts. During review, each candidate is either flagged for higher-resolution processing (and saved) or discarded as a false positive and the disk space reclaimed (see Figure 1 for an overview of the average data volumes per job at different processing stages). The desire to provide analysts with a streamlined method for this review process is at the very core of our design.

Zoom In Zoom Out Reset image size

Similarly, the network bandwidth constraints of the host led us to a data transfer configuration that focused on metadata rather than requiring the complete transfer of raw, unreviewed, and possibly spurious detection data over the Internet. Instead, metadata sufficient to describe the salient characteristics of a candidate event to a trained analyst was transferred into our candidate review framework. This careful selection process had the beneficial side effect of greatly limiting the size of the transferred products, allowing for a considerably longer retention period on the $\sim 10$ TB archive hosted at JPL.

Finally, security constraints were also critically important to the design, particularly because the system spans two separate security domains: NRAO and JPL. To comply with the security requirements of the host system, data transfer was configured on the NRAO system to allow read-only operations and was made accessible only to clients originating from the JPL domain. Furthermore, on the front-end, the functionality exposed by the web portal component interacted only with the local metadata archive, eliminating the possibility of corruption or inappropriate access to the raw observational data.

On the JPL side, the V-FASTR data products are processed through a metadata extraction and data archiving pipeline that eventually leads to the event candidates being available for inspection on the web portal. The pipeline is composed of three major software components: rsync, the OODT CAS Crawler, and the OODT File Manager, depicted in Figure 2.

• rsync. Data products are automatically transferred from the NRAO staging area to the JPL server using rsync. rsync is a popular application and data transfer protocol that allows to synchronize the content of a directory tree between two servers with minimal human intervention. It was chosen because of its simplicity, high performance, reliability, and wide range of configuration options. Through rsync, files are transferred in compressed format and using delta encoding, meaning that only the file differences are sent through subsequent transfers. For this project, an rsync server daemon was set up on the NRAO side to expose the data staging area where the products are collected. For security reasons, the daemon was restricted to allow read-only operations to clients originating from a designated JPL IP address. On the JPL side, an rsync client was set up to run hourly as a system cron job, transferring products to the JPL archive area. To minimize bandwidth usage, the client only transfers a very small subset of the data comprising a product directory tree, namely the detection images and the output and calibration files containing the metadata needed by the web portal. On average, this represents a reduction of the data product size by a factor of $3.5\times {{10}^{3}}$: from an average size of $\sim 35$ GB on the NRAO server (for a product with several detections), to $\sim 10$ MB on the JPL server. The rsync data transfer rates between the two servers were measured to be around $\sim 2$ MB s⁻¹, more than enough to transfer between 10 and 20 data products per day.
• CAS Crawler. Once the data products are transferred to the JPL server, they are automatically detected by the OODT CAS Crawler daemon, which runs at sub-hour time intervals to pick up new products as soon as they become available. The Crawler is responsible for notifying the OODT File Manager and therefore starting the product ingestion process. For this deployment, the Crawler was configured to send a signal only if two preconditions are both satisfied: (1) a similarly named product does not already exist in the File Manager catalog and (2) the product directory contains a special marker file indicating that the product has been processed by the mail program, and therefore is in a complete state (i.e., no files are missing).
• CAS File Manager. The OODT CAS File Manager is a customizable software component that is responsible for processing and archiving a data product, making it available for query and access to clients. For this project, the File Manager was deployed with the default Apache Lucene metadata back-end, and configured to archive products in-place, i.e., without moving them to a separate archive directory, otherwise the rsync process would transfer them again from the NRAO server. Additionally, we leveraged the extensibility of the OODT framework by configuring the File Manager with custom metadata extractors that were purposely written to parse the information contained in the V-FASTR output and calibration files. Information is extracted at the three levels that comprise the hierarchy of a V-FASTR data product: job, scan, and event. Additionally, a numerical algorithm was written to assign each pair of available images (-det.jpg and -dedisp.jpg) to the event that generated them.

Zoom In Zoom Out Reset image size

In general, a File Manager can store metadata in its back-end catalog as different object types. Each object type is defined to contain multiple metadata fields, where each field is composed of a named key associated to one or more string values. For this project, the decision was made to maintain a one-to-one correspondence between a data product and the corresponding metadata ingested into the catalog. So rather than defining three object types for jobs, scans, and events, a single object type was used holding all information for a data product in a single container, with dynamically named keys that are encoded to contain the scan and event numbers. This decision was motivated by the desire to simplify and optimize the querying of information by the web portal client, since all metadata for a product is retrieved through a single request to the File Manager. As a consequence, the default Apache Lucene metadata catalog implementation had to be slightly modified to allow for the ingestion of dynamically named metadata fields.

The second major component of the candidate review framework is an interactive web portal. The primary purpose of the portal is to provide a convenient online environment for the location-independent perusal and assessment of potential candidates in context. The portal provides V-FASTR analysts with the ability to quickly navigate through the available information to identify candidates worthy of further inspection on a familiar web platform.

The portal has been implemented as a PHP web application using the Apache OODT Balance web framework running on top of the Apache HTTPD Web Server. OODT Balance was chosen here for its ability to easily integrate with the OODT components in the back-end metadata pipeline, namely the OODT CAS File Manager described earlier. Furthermore, the flexible, modular approach of the framework allowed us to quickly connect the web portal to the metadata repository and rapidly begin constructing the necessary views specific to the V-FASTR candidate review and validation use cases.

As Figure 3 shows, the web portal offers a variety of views of the available metadata which are hierarchically organized to match the conceptual relationships in the data. At the highest level, a job or run might consist of multiple scans, each of which may itself contain multiple detection event candidates. This hierarchy, expressed in the metadata, is preserved in the layout of the portal views, and the breadcrumb navigation provided to facilitate orientation within the nested structure.

Zoom In Zoom Out Reset image size

At the level of an individual event candidate (Figure 3, middle image), two graphical representations of the event are available to assist analysts in classifying the nature of the signal. These images are generated automatically as part of the initial candidate identification process (Wayth et al. 2011), and they provide a trained analyst the necessary structural clues needed to rapidly assess the received signal as being genuinely extraterrestrial in origin or merely a product of RFI.

To support both metadata browsing in context and the desire for an analyst to be able to rapidly peruse the image representations of an entire job (many events in many scans) at once, a compromise was struck whereby, for each job, a portal user may select a traditional, hierarchical navigation or a flattened view in which all of the (possibly numerous) event candidates are presented simultaneously on screen and can be accessed serially simply by scrolling the view.

Together, the metadata pipeline and the web portal constitute an end-to-end framework for capturing, archiving, and presenting metadata about detected transient event candidates to V-FASTR scientists. Furthermore, by providing a reliable process and flexible interface, the system directly streamlines the analysis process, boosting the overall efficiency of the project.