Seminar: From Cloud to Edge: Building Instruments for Today’s Science

Abstract

For better or worse, access to scientific instruments shapes a field: to validate hypotheses we need an instrument where we can deploy, capture, measure, and record relevant phenomena. For systems experimentation in computer science – in other words, projects that build and evaluate new operating systems, improve energy efficiency, or investigate performance – this means first and foremost an instrument that supports deep reconfiguration allowing investigators to change everything from firmware, through operating systems kernel, to higher levels of the software stack. It also means access to a great diversity of state-of-the-art hardware, including the newest types of architectures, memory and storage options, as well as the newest GPUs, FPGAs, and other accelerators that serve as a platform for the developing AI revolution. And most recently, it means supporting scalable computing at edge and programmable networking such that it allows investigators to create interesting experiments in the edge to cloud continuum. Most importantly though, a scientific instrument needs to support the capability to evolve as new scientific problems arise and require new features in an experimentation platform and as methodology approaches emerge and drive change.

In this talk, I will describe the status and evolution of two scientific instruments: Chameleon and FLOTO. Chameleon is a primarily datacenter-based virtual facility for computer science research. It consists of hundreds of diverse high-end nodes, distributed between core sites at University of Chicago, the Texas Advanced Computing Center (TACC) – and now also NCAR – as well as a few smaller volunteer sites. The hardware of this exploration platform is bare metal reconfigurable, allowing investigators to customize firmware, boot from custom kernel, as well as orchestrate complex distributed deployments connected using programmable networks. More recently, the system  also introduced features that allow investigators to create and program custom distributed deployments using edge devices (CHI@Edge). Over almost ten years of its operation, Chameleon has served 10,000+ users, working on 1,000+ research and education projects. Collectively, this community has produced 700+ research papers (that we were able to find).

FLOTO is an observational instrument that deploys 1,000 Raspberry Pis nationwide to measure broadband. It grew out of the exploration capability provided by Chameleon’s CHI@Edge to implement a large-scale, distributed observational instrument. By providing a simple deployment mechanism, FLOTO allows investigators to deploy edge devices (Raspberry Pis) easily, and manages diverse and evolving suites of broadband tests on those deployments. Over the last year or so, FLOTO has been deployed in various communities of interest, over 17 national and local network providers, and across different access technologies, such as fiber, cable, satellite, and fixed wireless. These deployments generated millions of datapoints that are publicly available and have been to support both computer science and policy research, supporting both research publication and policy decision making. Its model has been adapted to support investigations other than broadband measurement and expanded to serve the needs of National Discovery Cloud for Climate (NDCC) applications.

*Staff can find the event information on the Staff-Only events calendar.
External attendees can contact Lisa Larson for an invitation.