KM3NeT is a large research infrastructure that will consist of a network of deep-sea neutrino detectors in the Mediterranean Sea with user ports for Earth and Sea sciences. As a partner experiment in the European Horizon 2020 initiative projects ASTERICS and KM3NeT-INFRADEV, the KM3NeT collaboration strives to foster the dissemination of scientific results to the scientific community and to citizen scientists. We will report on the first steps that have already been taken towards this goal. Also, an overview of the next milestones will be given, as detailed in the KM3NeT data management plan. The plan is based on FAIR data management guidelines and contains, amongst others, information on the open data access standards to be employed.
The LIGO/Virgo collaboration features an open science programme, the goal of which
is to release data to the public, while providing the information and the tools necessary
to understand and use the data. I will review the past and current activities related to this open science programme.
11:05 Alberto Micol
(ESO): Publishing astronomical data: an ESO perspective [slides]
I will report on recent efforts to create an open data specification for
imaging atmospheric Cherenkov telescopes and the interplay of such a work
with the design of high-energy modular open source analysis frameworks. Its
usage should serve several purposes: enable the joint analysis of data from
several observatories, be a vehicle for open data releases and legacy
projects, and ease the checks on the reproducibility of results.
I will discuss, in this context, prototyping activities to export data into
this format and the inclusion of the first real TeV data release from
H.E.S.S. This data release can be seen as a test bench for the gateway
prototype, based on IVOA standards, that is developed at the Paris Data
Analysis Centre (PADC) in the context of the Cherenkov Telescope Array
(CTA). This prototype is a web client for query and visualisation that
integrates the IVOA Provenance data model, as well as Authentication &
Authorization management.
I will finally present the specific needs and challenges of the CTA project
concerning provenance information, one of the most complex use case for the
definition of the IVOA Provenance Data Model.
Why are DOI still an issue in Astronomy? Although in a broader community context
(e. g., RDA)
the usage of DOI for data sets is fairly uncontested, the pickup in astronomy is limited,
and even within the IVOA the notion of DOI is indistinct. I would like to raise again the issue of using DOI for astronomical data sets.
13:30 Carlo Maria Zwölf
(Observatoire de Paris): Recent outcomes of the Virtual Atomic and Molecular Data Center (VAMDC) in data publication, interoperability and standards [slides]
We present the most recent outcomes of the Virtual Atomic and Molecular Data Center in these main fields:
Current Challenges in data publication and publication tools: We give an overview of the VAMDC Query Store (which is an implementation of the Data Citation RDA recommendation) by highlighting, in particular, the new DOI functionality: this makes it possible to automatically obtain DOIs for any data extracted from TAP services when a given user wishes to cite these data in publications.
Developments to facilitate date exploitation by astronomers: We present the renewed version of the SpectCol analysis tool (allowing to cross-match in an interoperable way, with spectroscopic and collisional data coming from heterogeneous databases), and a data conversion web service to convert any molecular data extracted from VAMDC into the widely used Hitran format.
Use and adoption of Virtual Observatory standards: After recalling the work we are doing for convergence between the IVOA-SLAP standard and the VAMDC-XSAMS one, we present a first prototype service for extracting SLAP-formatted data from the VAMDC infrastructure.
About VAMDC
The Virtual Atomic and Molecular Data Centre Consortium (VAMDC)
is a consortium of institutes and research institutions that
share a common technical and political framework for the
distribution and curation of atomic and molecular data.
From the technical point of view, VAMDC federate 30
heterogeneous databases (including VALD, Hitran, JPL,
CDSM, Basecol, etc…) into an interoperable e-infrastructure.
VAMDC is built adopting IVOA components (registry,
TAP interfaces for discovering data, SAMP connectors
for data communication between post-processors).
SPECTCOL is a graphical tool implemented in Java. It allows to manipulate and combine spectroscopic and collisional data coming from the databases (BASECOL, CDMS, HITRAN, JPL,…) using VAMDC technology.
SVOCat is an application intended to facilitate the publication of an astronomical catalogue, both as a web page and as a Virtual Observatory ConeSearch service, while also implementing the SSA and DataLink protocols where appropriate.
Set in motion by the IVOA, the publication of astronomical research data has gained more and more importance over the years. Big collaborations and traditional research projects alike have
recognised the additional value of publishing their data. Many of these projects offer a dedicated website, which they also use to publish their data releases. Sadly, most of these websites are tailor-made for their particular use case and are therefore not transferable to future projects.
At the Leibniz Institute for Astrophysics Potsdam (AIP) we gained profound experience with both the maintenance and the development of such applications, and we have gained insight on both the dark and the bright sides of this approach. Based on these experiences, we use the Daiquiri framework to host a number of sites, e. g., the RAVE survey, the CosmoSim database, APPLAUSE (Archives of Photographic PLates for Astronomical USE), and the Gaia@AIP services.
Daiquiri is developed and maintained by the E-Science group at AIP. The Daiquiri software stack has now been released in its second incarnation based on the Python Django framework. The data sets are stored in SQL databases and published either to the public, or for a restricted groups of researchers. Access is possible via an interactive SQL query interface, the TAP protocol, or an IVOA Cone Search. The query parser package is part of the Daiquiri framework and allows to query the data using ADQL as well as the native backend SQL language. Currently, MySQL and PostgreSQL are supported.
Each user has her / his own private tablebase where the results of all queries are stored and can be retrieved and downloaded at any time. Along with data access, Daiquiri supports user management, contact messages, and meeting management, as well as an integration of Wordpress for documentation and project presentation. Daiquiri consists of a common code base, which is centrally maintained, and site-specific applications, which are highly customizable to project requirements. Daiquiri is open source software and is licensed under the Apache License 2.0. The source code is available on GitHub: https://github.com/aipescience/django-daiquiri
Aladin Desktop, Aladin Lite, Sitools / MIZAR, Stellarium, KStars, Firefly: more and more astronomical tools are adopting the IVOA HiPS system (Hierarchical Progressive Survey). This talk will present the HiPS history and its mechanism. And, concretely, what is required to transform an image collection into a HiPS and how to publish it for these viewers.
15:45 Christian Dersch
(Philipps-Universität Marburg): The benefits for astronomical software from integration into a Linux distribution: The case of Fedora Astronomy [slides]
Fedora Astronomy is an effort to improve the astronomical software ecosystem. Many fundamental software packages in astronomy are free software and are required for the astronomers' daily work. An example is the Astropy package with its many affiliated packages for more specialised applications, but also the AstrOmatic software, such as SExtractor, and applications to access the Virtual Observatory. Fedora Astronomy is organized as a Special Interest Group (SIG) that takes care of the integration of astronomical software into the Fedora Linux distribution. The advantage of this way of software distribution is a good integration into the operating system, and also between the different packages. Another task of the Fedora Astronomy SIG is the deployment of a ready-to-use Linux environment for the purposes of both amateur and professional astronomers, the Fedora Astronomy Lab.
Finally, the Fedora Astronomy SIG discusses issues with upstream developers, for example changes in depending applications or licensing, and takes care of the cooperation with other distributions' projects such as Debian Astronomy.
The talk covers the organization of the Fedora Astronomy SIG as well as the current state of Fedora Astronomy and its position in the astronomical software ecosystem. Furthermore, possible improvements in scientific software distribution and schemes for cooperation between different Linux distributions are discussed.
The Space Science Data Center is an Italian facility of the Italian Space Agency, a multi-mission science operations, data processing, and data archiving center that provides support to several scientific space missions. The current status of the implementation activities related to VO techinques will be presented, then the potentiality of VO development will also be discussed.
09:40 Adriaan Renting
(ASTRON): LOFAR data formats and the Virtual Observatory [slides]
The LOFAR Radio Telescope supports many types of observations and data formats. I will present a short description of the telescope, what data formats we support, which ones of those are currently available in our archive and the problems we have encountered in trying to model and expose our data to the VO.
10:10 Marco Iacobelli
(ASTRON): The ASTRON data portal: enabling access and exploitation of radio data collections through the VO [slides]
Radio astronomical data collections are becoming available through the VO. However, data from a new generation of interferometric facilities pose challenges to standard VO tools that need to be faced in order to properly expose scientific data products to the astronomical community. In this talk, I will overview the ongoing ASTRON / LOFAR efforts to adopt VO standards for its data collections and also the progress of relevant projects.
10:30 – 11:00
Coffee break
11:00 – 12:40
Plenary session, chair: Agata Trovato
11:00 Volodymyr Savchenko
(University of Geneva): The Astrophysical Online Analysis System, powered by a provenance data model Application to a "live" data archive and real-time analytics of multi-messenger transients [slides]
We present the Online Analysis System (OAS) platform, facilitating reduction and analytics of astronomical data. The version planned for public release in the coming months allows to effortlessly explore and exploit observations of ESA's INternational Gamma-Ray Laboratory (INTEGRAL). The platform can be accessed through a simple on-line interface, suitable for researchers not familiar with INTEGRAL data reduction but interested in astrophysical phenomena.
A more evolved analysis configuration can be achieved through detailed formulation of the service requests through an HTTP API. Our goal is to combine both quick-look in-browser analysis (providing robust but simple results) and fine-grained control through the API for domain experts (allowing interoperability and integration in federated data analysis workflows).
The analysis results at different degrees of reduction are archived as static products, annotated and indexed with the respective provenance graphs. This structure allows to follow the lineage of the scientific products from the raw spacecraft data to the final scientific publication. These products can be referenced with unique identifiers, derived from their provenance, suitable for assigning citable DOI (e. g., through Zenodo). These references address statically available archive products, while at the same time referring to the online analysis platform, allowing to trace every analysis step, and to re-execute the pipeline, reproducing the result from scratch, optionally introducing changes that are testing the robustness of the result. Wherever possible, we will provide an interface conforming to the VO standards for astronomical data products.
A simple, immediate, automated, and reliable interface to the diverse astronomical data has proven to be most important in multi-messenger time domain astronomy. We discuss the services that have been developed for promptly sharing INTEGRAL data for transients, which were exploited in the recent observations of different multi-messenger transients. Finally, we discuss the integration of these services with the OAS platform.
The Euclid satellite is an ESA mission scheduled for launch in 2020. It will observe an area of 15,000 deg2 with two instruments, the Visible Imaging Channel (VIS) and the Near IR Spectrometer and imaging Photometer (NISP). Ground-based imaging data in griz from surveys such as the Dark Energy Survey or KIDS complement the Euclid data to enable photo-z determination. The mission investigates the distance-redshift relationship and the evolution of cosmic structures by measuring shapes and redshifts of galaxies and clusters of galaxies out to redshifts of about 2.
Processing the vast amount of Euclid data from raw images to object catalogs, to photometric redshifts and weak lensing, and eventually to scientific results poses large challenges for data processing and data distribution. To cope with those challenges, the Euclid Science Ground Segment is organized in nine Organizational Units (OUs) that implement the software for the various processing steps and seven Science Data Centers (SDCs) that optimize and run those processing steps on their computer clusters.
To test the data processing, a sequence of Science Data Challenges are undertaken that run Euclid Processing functions on simulated input data, which is based on a mock universe with realistic fluxes and morphologies from empirical descriptions.
We present the status of the Euclid project, discuss the challenges that were encountered so far, and show the latest result of the Science Data Challenge 4/5/6, which tests our processing down to the weak lensing and the spectral analysis.
The ALMA observatory has already delivered huge amounts of data. Those data are accessible to download via the ALMA science archive portal from their parent project id. Here we present ARTEMIX, a development from the Paris Observatory that aims to provide a quick-look access to the ALMA datacubes as well as a synthetic view of the observation and imaging parameters (positions / frequencies).
As a network of deep-sea neutrino telescopes in the Mediterranean, the KM3NeT research infrastructure will provide significant real-time capabilities for the detection of high energy astrophysical neutrinos and low energy supernova neutrino bursts. Within this context, automated multi-messenger analyses will allow a prompt release of information to the astroparticle community in case of detection. At the same time, external alerts can be exploited to improve the KM3NeT sensitivity to transient sources. These features will be fundamental for the integration of the experiment in the existing and future global alert networks, as well as for the development of a rich multi-messenger program within the ASTERICS and KM3NeT-INFRADEV H2020 initiative projects.
The IVOA has proposed a framework to describe provenance metadata for the dissemination of astronomical data collections. This effort re-uses concepts proposed by the W3C Provenance Data Model and interprets those in the IVOA ecosystem. The main description elements are presented, together with their implementation strategies.
This talk will present the construction of provenance information for the third data release of the APPLAUSE database.
The APPLAUSE (Archives of Photographic PLates for Astronomical USE) project processes digitised photographic plates from various plate archives of the Hamburg, Bamberg, Tartu, and Potsdam observatories. The plates are digitized with high-resolution flatbed scanners. In addition, the corresponding plate envelopes and observation logbooks are digitized, and further metadata are available in the database. With the pyplate software package, positions and magnitudes of objects are extracted from the scans and calibrated against modern catalogs. The 3rd Data Release catalogue, scheduled for July 2018, contains over 3 billion sources.
APPLAUSE is available at https://plate-archive.org.
14:20 Markus Nullmeier
(GAVO / Uni Heidelberg): Effective searching in provenance graphs with custom ADQL functions [slides]
The
IVOA Provenance Data Model, currently in working draft stage, closely follows the quite general W3C (World Wide Web Consortium) recommendations for recording provenance of data. This means that the ensemble of provenance information actually makes up a web of arbitrarily structured relations, i. e., it features the properties of a graph in the sense of graph theory. Thus, many meaningful queries about provenance such as "all persons that were directly or indirectly involved in creating a given image", "all images that depend on reductions from a certain calibration dataset", or "all intermediate processing steps between two datasets", may only be calculated via graph traversal, a feature not available in the IVOA ADQL query language. While in the long run ADQL should be probably extended with features from established graph query languages such as openCypher, we here explore the suitability for graph queries of presently available means for ADQL extension, namely, so-called "user"-defined functions. Using (recursive) "common table expressions" (CTEs) that are available in virtually any SQL database commonly used in astronomical data centres, graph queries prove to be a feasible addition to existing ADQL implementations. Next to showing concrete examples, we discuss the benefits of using genuine graph databases, such as
Bitnine Agens Graph.
This talk reviews the current status of VO protocols for discovery, description, and access of data 'cubes' or 'hyper cubes', with a focus on implementation tricks. It will also give the status of the ongoing work for time series.
15:30 – 16:30
Concluding discussion of the meeting's results[slides]
Friday is reserved for hands-on consulting sessions on data publication.
Please try to specify your consulting needs for this day while
registering, or via e-mail.
Stefan Becker, François Bonnarel, Marco Molinaro, Dave Morris, Carlos Rodrigo, [Markus Demleitner]
Look at the plan on the venue page page in order to locate
the parallel sessions.
09:30 – 10:30
Carlos Rodrigo "Naïve" tutorial: detailed and oversimplified — but valid
— development of a VO Simple ConeSearch Service
from scratch
(plus SVO publication tools if time allows)
All others Pre-arranged consulting sessions
10:30 – 11:00
Coffee break
11:00 – 12:30
Parallel hands-on sessions
Stefan Becker Consulting on AstroPy, PyVO and ADQL
François Bonnarel Consulting on IVOA DAL and DM specifications
Marco Molinaro Chat on data resource publishing
Dave Morris Listening to your requirements, or providing
help with, publishing time-domain, time-series, and VOEvent data
Carlos Rodrigo VO consulting, including SVO tools
12:30 – 13:30
Lunch
13:30 – 14:30
Parallel hands-on sessions
Stefan Becker Consulting on AstroPy, PyVO and ADQL
François Bonnarel Consulting on IVOA DAL and DM specifications
Marco Molinaro Chat on data resource publishing
Dave Morris Listening to your requirements, or providing
help with, publishing time-domain, time-series, and VOEvent data
Carlos Rodrigo VO consulting, including SVO tools
14:30 – 15:00
Coffee break
15:00 – 16:00
Parallel hands-on sessions
Stefan Becker Consulting on AstroPy, PyVO and ADQL
François Bonnarel Consulting on IVOA DAL and DM specifications
Marco Molinaro Chat on data resource publishing
Dave Morris Listening to your requirements, or providing
help with, publishing time-domain, time-series, and VOEvent data