ICRAR’s Data Intensive Astronomy program’s main focus is the development of the data management and processing technologies required for the SKA. The program is divided into four streams –
- Science Data Processor (SDP) Pre-construction (part of the SKA Pre-construction program)
- Survey Science Support
- System Support
- Operational Systems Support
The ICRAR DIA team, located at the Centre’s UWA node, is comprised of researchers with a background from astronomy and industry who have been involved and led the development of data and operations systems for billion-Euro astronomical infrastructure in Europe and South America as well as industry projects in various domains. The team is involved in many, very diverse projects, including
- The Science Data Processor (SDP) Consortium of the Square Kilometre Array.
- The CHILES project
- The Murchison Widefield Array (MWA)
- The ASKAP DINGO survey project
- The ASKAP WALLABY survey project
- The International Virtual Observatory Alliance (IVOA)
As part of these projects the team is developing software and systems as well as expert systems (machine learning) and also provide support for operational systems, including the in-house ICRAR compute lab and the 20+PB MWA archive. We are also very heavy users of the Pawsey Supercomputing Centre as well as a number of other supercomputers around the world.
We do maintain active collaborations with many other academic and non-academic institutions both in astronomy, but also in other fields. In addition we are running a number of projects with industry partners from small startups to globally operating companies. Our group of students, which range from summer students, over Masters to PhD students are working on challenging and cutting-edge projects in pure computer science and cross-over between computing and astronomy. We are always open to discuss new ideas and trends in computer science and astronomical data reduction and software and are running teaching and training, industry and science workshops both in-house and externally.
Prior to 2014, the DIA program was called Information and Communication Technology program, but that was deemed to be a too generic term providing no focus. ICRAR’s ICT program focused on developing core capabilities through development of a Data Intensive Research Pathfinder, Conceptual Design Studies for the SKA and High Performance Computing (supercomputing). The Data Intensive Research Pathfinder involved the implementation and commissioning of dataflow and data archiving infrastructure for the Murchison Widefield Array (MWA). The MWA archive has been fully operational since July 2013 and is receiving data from the array at a rate of up to 1000 MB/s. The system provides data migration to various partner institutions around the world at the same rate. In addition it currently serves user requests at a rate exceeding the input data rate. Implementation of the MWA archive helped ICRAR secure leadership of the Data Layer task in the Science Data Processing (SDP) Consortium for the SKA.
Meet the Data Intensive Astronomy Team
Case Study: Think Bottom Up
Data Intensive Astronomy Focuses
Science Data Processor (SDP) Pre-construction
The Science Data Processor will develop all the necessary infrastructure to produce scientific data from the raw signals of the SKA’s radio correlators. ICRAR is a member of the SKA SDP Consortium and is leading the data element of the project, which involves identifying data flow and data life-cycle requirements and creating and verifying prototype solutions. We are also assisting five other work streams – system engineering, architectural design, computing platform, local monitoring and control and project management.
Survey Science Support
Survey Science Support broadly aims to optimise the productivity of data intensive science projects, including the SKA. The project investigates the feasibility of candidate Science Data Processor solutions for the SKA, linking it strongly to the SDP Pre-construction project. Part of this involves working with collaborators to investigate cloud computing solutions for processing high volumes of data. The Survey Science Support project also supports theSkyNet (crowd-sourced computing power campaign) and a number of current and future surveys including GLEAM (MWA), DINGO (ASKAP), WALLABY (ASKAP), GAMA (Multi-Wavelength) and CRAFT (ASKAP).
ICRAR’s Data Intensive Astronomy team also supports the maintenance of computing infrastructure at our UWA node, including the servers, the server room, and the storage servers, plus auxiliary equipment such as the network connection between servers at the Pawsey Centre and UWA.