data

A prototype field-to-publication data system for a multi-variable permafrost observation network.

A prototype field-to-publication data system for a multi-variable permafrost observation network.
Blog post by Nick Brown, NSERC PermafrostNet Data Scientist

Analysis and prediction of permafrost change are hampered by lack of observational data. In collaboration with Stephan Gruber, Peter Pulsifer, and Amos Hayes, we developed a permafrost data management system to support permafrost observation networks that involve many different kinds of permafrost data.

We identify five broad challenges for permafrost data management and publication: (1) existing data management strategies do not scale well, (2) data users have different skills and needs, (3) permafrost data are varied, (4) resources for permafrost data management are limited, and (5) existing permafrost data sources are difficult to integrate. Our prototype system supports a permafrost data workflow from observation to the distribution of interoperable data. The system simplifies data publication and management, although we identify and discuss several hurdles in adapting the CF conventions and ERDDAP for permafrost data. Our learning can inform organizations who collect, manage, or distribute permafrost data or those who manage large observation networks.

In summary:

  • Five broad challenges limit permafrost data management and publication.
  • We frame these challenges as requirements, and identify similarities with the FAIR principles.
  • We developed a prototype a permafrost data system to support field-to-publication workflows.
  • In this project, we use an “adopt and adapt” approach for standards and software.
  • Our data system supports more FAIR permafrost data.

Nicholas Brown, Stephan Gruber, Peter Pulsifer, Amos Hayes, A prototype field-to-publication data system for a multi-variable permafrost observation network, Environmental Modelling & Software, Volume 175, 2024, 106006, ISSN 1364-8152, doi:10.1016/j.envsoft.2024.106006

This research was enabled in part by support provided by Compute Ontario et du Digital Research Alliance of Canada.

By nick brown, ago
Global map summarizing locations of field sites where electrical resistivity tomography (ERT) has been used to study permafrost (2000–22) based on the literature search.

Best practices for using electrical resistivity tomography to investigate permafrost.

Best practices for using electrical resistivity tomography to investigate permafrost

Une étude récente de Teddi Herring suggère des moyens d'améliorer la façon dont la tomographie par résistivité électrique (Electrical Resistivity Tomography ou ERT) est utilisée pour le pergélisol et met en évidence les progrès récents dans cette approche. L'ERT est une technique extrêmement utile pour étudier le pergélisol, car elle nous permet de connaître la profondeur de la couche de pergélisol et d'identifier les zones contenant de la glace.

Le nombre de publications d'études utilisant l'ERT pour analyser le pergélisol a été multiplié par 10 au cours des 20 dernières années, et bien que des défis demeurent et qu'il n'existe pas encore de « meilleure façon » unique de le faire, l'étude formule des recommandations pour mener des études ERT afin de maximiser l’utilité des données existantes et futures.

Herring T,  Lewkowicz AG,  Hauck C, et al.  Best practices for using electrical resistivity tomography to investigate permafrost. Permafrost and Periglac Process.  2023; 34(4): 494-512. doi:10.1002/ppp.2207

Global map summarizing locations of field sites where electrical resistivity tomography (ERT) has been used to study permafrost (2000–22) based on the literature search.
Carte mondiale résumant les emplacements des sites de terrain où la tomographie par résistivité électrique (ERT) a été utilisée pour étudier le pergélisol (2000 à 2022) sur la base de la recherche documentaire.

By Tristan MacLean, ago

Formats and variables – making permafrost data more useful

by Nick Brown, NSERC PermafrostNet Data Scientist

Students in NSERC PermafrostNet’s Theme 1, led by postdocs Michel Paquette and Samuel Gagnon, have digitized 2 historic reports (Hodgson, 1982 et Hodgson, & Nixon 1998) containing profile data from more than 250 boreholes in polar desert environments. Information on ground ice content, ground ice descriptions, soil type, grain size and more, is now available for re-use in a variety of formats using the PermafrostNet ERDDAP data server. Data can be accessed and searched from a single borehole, or from the entire report. Both the data et metadata can be downloaded as a csv.

These data take advantage of the new additions to the CF standard name list to make the data more interoperable and reuseable. Nick Brown has also documented a set of standardized variable names and their associated CF Names – Standardized permafrost variable names and equivalent CF Names. This technical note document describes the variables that are on the NSERC PermafrostNet ERDDAP and their associated metadata. It can also be used as a guide to connect commonly-used permafrost variables to the recommended CF standard_name.

One of the steps involved in redistributing these data on the ERDDAP server is converting them into the NetCDF data format. This is a way of distributing scientific data that is gaining popularity because it is self-documenting and uses a several well-established community-built standards to increase data interoperability. These include the CF conventions and the Attribute Convention for Data Discovery (ACDD). There are a number of data distribution platforms (ERDDAP, THREDDS, HYRAX) that can be used to easily distribute NetCDF files using a standardized API and web interface. NSERC PermafrostNet uses ERDDAP as a data distribution platform for these reasons and because it offers a graphical web interface for those who want to access data interactively. It also makes it possible to adopt the associated data standards as one way to increase the interoperability of permafrost data generated by and used within the network.

The NetCDF format and associated standards were originally developed for atmospheric, oceanographic and climatic data, so there are few examples of these files being used to represent observed soil profiles or geotechnical borehole data, and the CF standard_name table lacks many variables that are relevant to permafrost science. A new technical document – Representing geotechnical borehole profile data with netCDF and ERDDAP provides considerations and recommendations for structuring observed borehole profile data in a way that complies with existing standards, is distributable on ERDDAP, and useable by permafrost scientists.

These new documents are available on the PermafrosNet Data Standard Recommendations page.

By nick brown, ago

Take your permafrost metadata to the next level

by Nick Brown, NSERC PermafrostNet Data Scientist

Everyone’s been there. You are about to publish your dataset, but you’ve just spent the last half-hour trying to decide on the best formatting and naming conventions. Should that column be named “ground temperature”, “soil_temperature”, or just “Tg”? Maybe you should include units too; “ground_temperature_degree_C” has a ring to it. But how will other people know exactly what your dataset represents? Good metadata is the answer.

You may not realize it, but there are resources to help you in this situation. The CF Standard name table is a curated list of terms used to unambiguously identify the kinds of measurements in a dataset. For self-describing formats like netcdf, you can include these attributes directly in your data file, following the CF standards. Or, if you are publishing your data in a text file, like a CSV, you can include the information about each column in your dataset in a separate metadata file. This way, the name of your data column doesn’t have to do so much work describing itself.

Michel Paquette, Theme 1 Post-doctoral Fellow (Université de Montréal, NSERC PermafrostNet) and I have coordinated the inclusion of 12 additional permafrost-related terms (see table below) in the latest release of the Standard Names vocabulary (version 78). These terms will be particularly useful for field scientists wanting to make their published datasets more interoperable and for data publishers who host permafrost-related data. An additional 14 terms relevant to permafrost science have since been requested. The addition of permafrost-related terms contributes to improved data interoperability with the atmospheric science and modelling communities, where the CF terms are widely used.

For an up-to-date list of permafrost-related terms added to the CF names list by NSERC PermafrostNet, visit the Data Standard Recommendations page.

Common nameCF NameStatus
Ground temperaturetemperature_in_groundAdded (v78)
Excess iceice_volume_in_frozen_ground_
in_excess_of_pore_volume_in_unfrozen_ground_
expressed_as_fraction_of_frozen_ground_volume		Added (v78)
Ice saturation indexratio_of_ice_volume_in_frozen_ground_
to_pore_volume_in_unfrozen_ground		Added (v78)
Thaw depthdepth_at_base_of_unfrozen_groundAdded (v78)
Frozen thermal conductivitythermal_conductivity_of_frozen_groundAdded (v78)
Frozen heat capacityspecific_heat_capacity_of_frozen_groundAdded (v78)
Gravel content by massmass_fraction_of_gravel_in_soilAdded (v78)
Sand content by massmass_fraction_of_sand_in_soilAdded (v78)
Silt content by massmass_fraction_of_silt_in_soilAdded (v78)
Clay content by massmass_fraction_of_clay_in_soilAdded (v78)
Organic matter content by massmass_fraction_of_organic_matter_in_soilAdded (v78)
Soil water pHsoil_water_phAdded (v78)

Permafrost related terms added to v78 of the CF Standard Name vocabulary by NSERC PermafrostNet

By nick brown, ago

Appel aux évaluations d’experts ! La carte des conditions de la glace de sol. Call for experts – The Ground Ice Map of Canada

“Ground ice maps will ever only be as good as the data behind them.”

Michel Paquette, NSERC PermafrostNet Post-doctoral Fellow, Université de Montréal

Appel aux évaluations d’experts !

Nous cherchons à obtenir les commentaires et la validation des experts de la communauté canadienne du pergélisol sur la récente carte de la glace de sol du Canada.

La carte des conditions de la glace de sol (ground ice map of Canada – GIMC) (Geological Survey of Canada, Open File 8713) présente une nouvelle cartographie à l’échelle nationale des conditions de glace de sol au Canada. L’évaluation du GIMC s’inscrit dans le cadre des efforts continus visant à fournir un retour d’information général et à évaluer des domaines spécifiques du GIMC, ce qui contribuera à la production de la première version de la carte de base du potentiel de glace de sol et des conditions géotechniques du pergélisol au Canada (GRIPv1) par PermafrostNet du CRSNG. Nous aimerions connaître votre avis d’expert sur l’exactitude de la GIMC actuelle. Votre expertise est précieuse, car les connaissances sur la glace de sol sont plutôt rares; elles dépendent fortement des connaissances du terrain et sont donc dispersées dans toute la communauté du pergélisol.

Nous aimerions connaître votre avis d’expert. Veuillez visiter notre page web pour plus d’informations et pour accéder à notre questionnaire – GIMC page web.

Call for expert evaluations!

We are looking to get expert feedback and validation from the Canadian permafrost community on the recent ground ice map of Canada.

The Ground Ice Map of Canada (GIMC) (Geological Survey of Canada, Open File 8713) presents new national-scale mapping of ground ice conditions in Canada. Evaluation of the GIMC is part of the ongoing efforts to provide general feedback, and evaluate specific areas of the GIMC, which will inform the production of the first version of the Ground Ice Potential and geotechnical permafrost base map of Canada (GRIPv1) by NSERC PermafrostNet.

We would like expert opinions on the accuracy of the current GIMC. Expertise is highly valued, as knowledge on ground ice is quite rare and highly site dependent, and therefore scattered all across the permafrost community.

Read more about it on our GIMC page.

By Tristan MacLean, ago

The 2020 NSERC PermafrostNet Data Hackathon

“The event was a great opportunity to meet some of the new network personnel and work together to make progress on some of their data-related challenges. As PermafrostNet continues to develop I hope we can have more events like this.”

Nick Brown, NSERC PermafrostNet Data Scientist

The first NSERC PermafrostNet Data Hackathon was held on August 19th, 2020, with fifteen researchers and data scientists getting together on Zoom to share best practices and discuss their latest challenges with data and coding.

PermafrostNet Hackathon Zoom

The event was organized to support members of the network in creating metadata records and publishing datasets.

The event was kicked off with a presentation by Nick Brown (NSERC PermafrostNet Data Scientist), who outlined the plan for the days activities and presented an introduction to the resources available through NSERC PermafrostNet. You can download the Hackathon presentation here.

The day was broken up into sessions for group work in breakout rooms, discussions and Q+A opportunities for the participants. There was plenty of opportunity for everyone to network and talk with new members of the network and experienced researchers about the varied data challenges they faced. Many of the participants were new members to NSERC PermafrostNet, and in some cases permafrost data work. It was this opportunity to find people to share data issues and ask questions that was found to be the most valuable aspect of the Hackathon by the participants.

You can take a look at the day’s schedule and further details about the event on our 2020 Hackathon page.

By Tristan MacLean, ago

Third Canadian Polar Data Workshop

“The workshop was the ideal venue to introduce PermafrostNet to the polar data management community and to learn from other communities of practice to ensure that the network will help permafrost data fit into the broader ecosystem of interoperability.”

Nick Brown

Le sous-thème Third Canadian Polar Data Workshop was held 18-21 February 2020 in Banff, Alberta. The event is organized by the Canadian Consortium for Arctic Data Interoperability (CCADI), who aim to advance collaboration, through development of an integrated Canadian arctic data management system and Arctic Research Data Infrastructure (ARDI). The purpose of the third Canadian Polar Data Workshop is to gather people involved in polar data management from across Canada to share ideas and identify key priorities and commitments for coordinating the work of the polar data community and, ultimately, advancing our work.

The workshop provided an opportunity for members of the Permafrost Network to discuss issues of data sharing architectures and data governance with research communities who are facing similar challenges. In attendance from NSERC PermafrostNet were scientific director Stephan Gruber, scientific committee member Peter Pulsifer, network data scientist Nick Brown as well as two members from the broader PermafrostNet community: Ashley Rudy (Northwest Territories Geological Survey) and Etienne Godin (Laval University).

Held at the Banff Centre, the workshop began with presentations from some of the attending organizations including PolarView, CCADI and Inuit Tapiriit Kanatami. There were presentations by network member Peter Pulsifer and a poster by the PermafrostNet data scientist Nick Brown on advancing interoperable and open permafrost data on day two. The agenda then turned to a more interactive format identifying the needs of the community for improved formal vocabularies and semantics, and visioning the formation of a committee to help coordinate polar data management. The event concluded with a hackathon focused on implementing code to make datasets more discoverable online. A relatively strong turnout by the permafrost community made for great conversations between formal sessions, and will help PermafrostNet make data interoperable with other initiatives.

Photos: Nick Brown

Third Canadian Polar Data Workshop Poster
Third Canadian Polar Data Workshop PosterTélécharger
By Tristan MacLean, ago