Datamart du SMC : Données météo ouvertes

& GeoMet : Accès à la météo en WMS et KML

Alexandre Leroux, M.Sc., Ing. Service météorologique du Canada Environnement Canada

Vision Géomatique 2014 12 octobre 2014

Environment Canada’sMeteorological Service of Canada

Authoritative dataThat’s the only place in Canada where numerical weather prediction (NWP) models are run in a fully 24/7 operational context

Canadian Meteorological Centre, Dorval

Page – 13-05-09

Meteorological Service of Canada’s

Open Data Server: The MSC Datamart

4

MSC Datamart: http://dd.meteo.gc.ca/

5

MSC Datamart — What it is

• Free and Open access for specialized users • Open data: freely available to everyone to use and republish as they wish,

the on the sole condition of attribution of the original data to Environment Canada

• Open formats: published specifications that can be used and implemented by anyone

• Target audience has some Weather and/or IT knowledge • Designed for direct, automated (batch) downloading of data via HTTP

protocol • Offer is complementary to EC’s weather.gc.ca website • Service is anonymous, supported on best-effort basis

• License • Give credit to Environment Canada • Commercial usage permitted • No username/password required

6

History

• 2004 • Initial rollout with MSC's GRIB data

• 2006 • AMIS (satellite service for the delivery of

meteorological alphanumeric data) ends; MSC Datamart used as alternative

• 2007-2013: • Addition of many other data types for

specialized users • 2014:

• Real-time notification service • Continuous improvement of offered

products

…

7

MSC Datamart — Statistics highlights

• Visitors and download volume: increased by factor of 7 since 2009 • 500,000 visitors daily • 14,000,000 downloads daily (includes directory listing requests)

• Popular products: • Citypages weather XML: 350M files/month • NWP model data in GRIB:

• NAEFS: 30M files/month • GDPS: 18M files/month • RDPS: 4M files/month

• Observations in SWOB-XML: 3M files/month • Most significant increases in past year: NAEFS Ensemble NWP

forecasts, Citypages weather, Weather warnings

8

Increasing use of the MSC Datamart as a dissemination channel:

Average number of files downloaded daily

9

Increasing use of the MSC Datamart as a dissemination channel:

Average data size downloaded daily

10

Who are the users?

• Public sector • Provincial agencies and departments, including Emergency

Management Organizations • State-owned companies • Municipalities

• Private sector • Weather companies • Mobile applications developers • Agriculture, transportation, energy, insurance, universities, etc.

• Hundreds of thousands of ordinary people… through third-party mobile and web applications

11

Popular datasets

• Citypage weather XML • Used directly by apps on mobile devices and other third-party products

• Numerical Weather Prediction forecasts model data in GRIB: • Global • Regional • Ensemble • HRDPS • Precipitation Analysis • Wave model

• Weather warnings in CAP-XML

• Weather observations in SWOB-XML (OGC O&M)

12

And there’s lots more available…

• Nowcasting data • Weather bulletins • Air Quality Health Index • Marine weather forecasts • Meteocodes • Hurricane tracks forecast • Weather forecasting regions • Vertical profiles, forecasted and observed • Diamondback Moth trajectories

13

Product example: Citypage XML

weather.gc.ca website MSC Datamart

14

Applications

Managing increasing NWP data volumes: NWP datasets provided daily on the MSC Datamart

Based on the 5TB of operational NWP input (including additional post-processing) we make available on MSC Datamart

90 GB of NWP data per day (1.8% of total daily NWP production)

Regional Forecast System

52 GB per day

Global Ensemble Forecast System

12 GB per day

Global Forecast System

9 GB per day 9 GB

per day

High Resolution 2.5km windows

Forecast System

… and other NWP models such as the Wave model, RDPA, etc

16

Open data formats currently offered

• Main data formats offered: • GRIB2 format — a World Meteorological Organization standard • Comma-separated values (CSV) • Extensible Markup Language (XML) • Raw text

• Other formats currently offered: • Esri Shapefiles • GIF images

17

User Interaction

• « Contact us » on the website

• Word of mouth

• dd_info mailing list: ~600 subscribers • http://lists.ec.gc.ca/cgi-bin/mailman/listinfo/dd_info

• Online documentation • dps-client@ec.gc.ca: bug reports, questions, requests

18

The AMQP Experiment

• Fact: the MSC Datamart is a “pull” service: to retrieve time sensitive data, users need to make repeat requests, which fail continually until new data appears

• Solution: notify users when the product they want is available, and give them the option to download

• The technological tool: the ‘Advanced Message Queuing Protocol’, a finance and banking industry high-rate message routing system (open standard, software libre)

• Win-win: less bandwidth used by both the user and MSC • Users are invited to use it since January 2014

19

Future Plans for the MSC Datamart

• Upcoming geospatial web services for MSC Datamart content => ‘GeoMet 2’

• Increased presence on Government of Canada’s Open Data Portal, open.canada.ca

• Improved metadata ISO19115-NAP for discovery and access • Assess AMQP real-time notifications experiment and make

recommendations • New potential datasets:

• HRDPS-Continental NWP model • Climate and hydrological datasets • Upper air analyses

• Caldas, Synthetic surface data

• “Store front” on ec.gc.ca / canada.ca

Page – 13-05-09

Meteorological Service of Canada’s

GeoMet Geospatial Web Services

MSC has been providing access to NWP model outputs data and static maps for a long time

but only specialized users could manipulate the data and use it...

until we began offering new interactive geospatial data access methods: WMS and KML

this is GeoMet!launched to the public in July 2013

Canadian weather layers on your smartphone, in your tools and your web maps?

you can now, with GeoMet

Web Map Service (WMS) and Network-KML?

- these are geospatial web services interoperable standards - interactive data visualization - can be accessed from various devices

web pages, client software, smartphones

What can WMS can do for you?

Capabilities - visualize data products and raw data - WMS is a web service, there’s no file download involved - layers are served on-the-fly, there’s no pre-generated images - data is sent only for a client-specified geographic extent - data is always the latest available - enable users to integrate the layers directly into their tools

how many html lines to build a website with an interactive map of GeoMet's weather layers?

lowering the barrier to consume MSC data

fully interactive web mapping website

- zoom, pan, query data

- control over multiple data layers

- overlay external data sources

- you can make your own map

24-lines html website showing the GDPS Rain (QPF) layer

GeoMet KML in Google Earth: easy access for everyone

HRDPS East - Air temperature

zoom in and GeoMet sends you higher resolution data

these layers can be animated

What’s in GeoMet for you?

Weather radar mosaic: - American and Canadian radar mosaic at 4km resolution, updated every 10 minutes, both 8 and 14-colors styles availableNWP models: - Global Deterministic Prediction System (GDPS) at 25 km resolution - Regional Deterministic Prediction System (RDPS) at 10 km resolution - High Resolution Deterministic Systems (HRDPS) at 2.5 km resolution - GEM - Modelling Air quality and Chemistry (GEM-MACH) model - Regional Deterministic Precipitation Analysis (RDPA) Layers include: - Temperature, precipitation, wind speed & direction, cloud cover, humidity, pressure and much more- Derived products such as: dominant precipitation type, wildfire forecasts, etc

GeoMet also serves its metadata in ISO19115

There’s even a new WMS server for NWP-Research experimental products, ‘RPN-WMS’: http://collaboration.cmc.ec.gc.ca/rpn-wms/?service=WMS&request=GetCapabilities

MSC data users are requesting this type of service

EMOs, industry, DND, the public, etc

«Overall we are very impressed with the capability and support the direction that you are taking with this

[GeoMet] initiative as it will be beneficial to many consumers of Canadian meteorological data.»

- Iain Russell, Director, Meteorological R&D, The Weather Network / Pelmorex Media Inc.

participants to the GeoMet Public Trial

GeoMet WMS allows sophisticated users to interact with MSC data into their tools

no file download

the data is always current data transferred only for the context-specific geographic extent

ArcGIS, one of plenty of software that supports the WMS standard

GeoMet WMS layers into Sécurité civile du Québec’s

G.O.LOC tool

This tool is used for emergency response (911, emergency measures, etc)

Only ~100 kilobytes of png tiles is transferred on-the-fly, no multi-megabytes file involved

GeoMet WMS layers into MASAS

MASAS: location-based situational awareness information and alerts for first response and emergency management agencies

Weather radar composite served in WMS

Another MASAS example

CHC’s hurricane and tropical storm forecast tracks from the GeoMet KML

Weather app for Windows tablets developed by Apption, relying on GeoMet’s WMS layers

In ArcGIS Online too

It’s mostly about enabling MSC clients to easily create products for their clients

GeoMet WMS

GeoMet KML

The Public

Accessing MSC weather layers

via third-party web maps, apps on mobile devices,

on Google Earth, and more!

Specialized users and Developers

Integration in their own tools, interactive web maps,

mobile apps, and more!

UberWeather.com

leveraging GeoMet to enhance their offer to the public

OpenWeatherMap.org

UrbanGEM Viewer for MSC’s CUDM system example show temperature layer, wind barbs and Google relief

colder air from the mountains goes around Bowen Island

interactive web-based interface to analyse weather model outputs

anyone from the public can build such tools now by using GeoMet

Build custom dashboards for your needs, which contain live GeoMet layers

Data Quality Dashboard

MetShow

Ontario Fire Program

GeoMet’s not only for maps!leveraging the WMS GetFeatureInfo command…

BigWaveDave.capunctual data from GeoMetto feed tables

nXstream Technologies

feeding graphs

Weather Vigilance

using readily available, widely used and mature technology and interoperable standards

It’s all about geospatial data dissemination Geospatial data is any data located in space: most of MSC’s data is geospatial data!

Opening the door to new possibilities - enabling our clients and partners to generate a new set of added-value products and services - easing location-based services, e.g. to provide integration with smartphones and mobile devices - streamlining MSC internal processes

if there’s only one line to remember

GeoMetversatile and efficient data dissemination method for displaying weather layers on interactive websites, mobile devices and client software using the WMS standard

GeoMet’s official page: http://www.ec.gc.ca/meteo-weather/default.asp?lang=En&n=C0D9B3D8-1(or just google ‘GeoMet Environment Canada’)

- Join the mailing list - Let us know about what you do with it

48

GeoMet Usage Statistics

• Serving about 100,000 maps daily during weekdays • No detailed usage statistics specifically tuned for

geospatial web services at the moment, but trend clearly show increasing interest and usage

49

Future Plans — GeoMet 2

• Lots more data, such as serving the MSC Datamart content via geospatial web services

• Offering the data in more formats and additional geospatial web services (e.g. GeoJSON, WFS, WCS, etc)

• Increasing MSC data’s discoverability and access on Canada’s Open Data Portal

Pour questions et commentaires : alexandre.leroux@ec.gc.ca