New Initiative to Help Unravel Cosmic Mysteries with Big Data

Open-source software will transform astronomers’ tools to analyze big data from Vera C. Rubin Observatory

25 August 2021

An expansive, multi-year collaboration will create new software to analyze the datasets from the upcoming Rubin Observatory Legacy Survey of Space and Time. The effort, led by Carnegie Mellon University and the University of Washington, will enable large-scale science fueled by the new data. The Community Science and Data Center at NSF's NOIRLab will collaborate with the team to make the software freely available to the entire astrophysics community.

The project, supported by ​​Schmidt Futures, will fundamentally change how scientists use modern computational methods to make sense of big data from Vera C. Rubin Observatory, a joint initiative of the National Science Foundation (NSF) and the Department of Energy (DOE), which is operated jointly by NSF’s NOIRLab and DOE’s SLAC National Accelerator Laboratory (SLAC). The primary mission of Rubin Observatory is to carry out the Legacy Survey of Space and Time (LSST), which will generate an unprecedented dataset for scientific research supported by both NSF and DOE. Scientists and software engineers from the Community Science and Data Center (CSDC) at NOIRLab will collaborate with teams from the University of Washington and Carnegie Mellon University to ensure the resulting analysis framework is accessible to the broader astronomical community.

“The software funded by this gift will magnify the scientific return on the public investment by the National Science Foundation and the Department of Energy to build and operate Rubin Observatory’s revolutionary telescope, camera, and data systems,” said CSDC Director Adam Bolton. "Through AURA’s membership in the LSST Corporation, we are able to connect with this project from its inception, which will allow us to engage NOIRLab users during the development phase and to support their use of these new tools in their future research."

​​Schmidt Futures is a philanthropic initiative founded by Eric and Wendy Schmidt. This project is part of Schmidt Futures’s work in astrophysics, which aims to accelerate our knowledge about the Universe by supporting the development of software and hardware platforms to facilitate research across the field of astronomy.

With the Legacy Survey of Space and Time, Rubin Observatory in northern Chile will usher in a golden age for time-domain astronomy by collecting and processing more than 20 terabytes of data each night — and up to 10 petabytes each year for 10 years — and will build detailed composite images of the southern sky. Over its expected decade of observations, astrophysicists estimate that the Rubin Observatory LSST Camera, funded by DOE and built at SLAC, will detect and capture images of an estimated 30 billion stars, galaxies, star clusters, and asteroids. The observatory will survey the entire visible southern sky every few nights and will essentially create a “time-lapse” movie of changes over different timescales to learn about the risks from asteroids, the nature of dark matter and dark energy, and many other phenomena in the changing Universe.

"Our goal is to maximize the scientific output and societal impact of the LSST, and these analysis tools will go a huge way toward doing just that,” said Jeno Sokoloski, Director for Science at the LSST Corporation. “They will be freely available to all researchers, students, teachers, and members of the general public."

Rubin Observatory will produce an unprecedented dataset through the Legacy Survey of Space and Time. To take advantage of this opportunity, the LSST Corporation created the LSST Interdisciplinary Network for Collaboration and Computing (LINCC), whose launch was announced on 9 August 2021 at the Rubin Observatory Project Community Workshop (login required). A primary goal of LINCC, jointly based at Carnegie Mellon University and the University of Washington, is to create new and improved analysis techniques that can accommodate the scale and complexity of the data, creating meaningful and useful pipelines of discovery for LSST data.

The support from Schmidt Futures will allow LINCC to build critical infrastructure and develop tools to help enable the large-scale science that will be fueled by the LSST data. The software will be made freely available to the entire astrophysics community. The LSST Corporation will run programs to engage the astrophysics community in the design, testing, and use of the new tools. CSDC will integrate the new tools within NOIRLab’s portfolio of research-enabling data services and make them accessible to all astronomers.

“Tools that utilize the power of cloud computing will allow any researcher to search and analyze data at the scale of the LSST, not just speeding up the rate at which we make discoveries but changing the scientific questions that we can ask,” said Andrew Connolly from the University of Washington.

Northwestern University and the University of Arizona, in addition to Carnegie Mellon and the University of Washington, are hub sites for LINCC. The University of Pittsburgh will partner with the Carnegie Mellon hub.

More information

NSF’s NOIRLab (National Optical-Infrared Astronomy Research Laboratory), the US center for ground-based optical-infrared astronomy, operates the international Gemini Observatory (a facility of NSF, NRC–Canada, ANID–Chile, MCTIC–Brazil, MINCyT–Argentina, and KASI–Republic of Korea), Kitt Peak National Observatory (KPNO), Cerro Tololo Inter-American Observatory (CTIO), the Community Science and Data Center (CSDC), and Vera C. Rubin Observatory (operated in cooperation with the Department of Energy’s SLAC National Accelerator Laboratory). It is managed by the Association of Universities for Research in Astronomy (AURA) under a cooperative agreement with NSF and is headquartered in Tucson, Arizona. The astronomical community is honored to have the opportunity to conduct astronomical research on Iolkam Du’ag (Kitt Peak) in Arizona, on Maunakea in Hawai‘i, and on Cerro Tololo and Cerro Pachón in Chile. We recognize and acknowledge the very significant cultural role and reverence that these sites have to the Tohono O'odham Nation, to the Native Hawaiian community, and to the local communities in Chile, respectively.

Rubin Observatory is a joint initiative of the National Science Foundation (NSF) and the Department of Energy (DOE). Its primary mission is to carry out the Legacy Survey of Space and Time, providing an unprecedented dataset for scientific research supported by both agencies. Rubin is operated jointly by NSF’s NOIRLab and SLAC National Accelerator Laboratory (SLAC).  NOIRLab is managed for NSF by the Association of Universities for Research in Astronomy (AURA) and SLAC is operated for DOE by Stanford University. 

Contacts

Adam Bolton
Director
NOIRLab's Community Science and Data Center
Email: adam.bolton@noirlab.edu

Bob Blum
Director for Operations
Vera C. Rubin Observatory, NSF’s NOIRLab
Tel: +1 520 318 8233
Email: bob.blum@noirlab.edu

Vanessa Thomas
Public Information Officer
NSF’s NOIRLab
Tel: +1 520 318 8132
Email: vanessa.thomas@noirlab.edu