SDI - PROJECT

FINAL REPORT
Using and sharing (near) real-time IoT data

Steffen Reichel
Saadat Tashbaeva
Christian Pleshberger
Kumushkan Abdimazhitova

Project on GitLab: https://gitlab.com/grmpfhmbl/16W_SDI_IP

INTRODUCTION

The aim of this project was to build an infrastructure, which collects environmental data from freely available
sources on the internet and make them available via an OGC standardized interface. For user-friendly access, the
goal was to create a web application that displays weather data in the area of Salzburg. It will be map-centred
and the user shall be able to select a monitoring station. This will show the user historic as well as the current
data.

The second objective was to create a notification service where the user can configure alarms for specific stations
when measured values meet a user defined threshold. This objective was dropped due to time constraints.

PROJECT ORGANIZATION

The project was split into four work packages with one person responsible for every work package. Each work
package contained several tasks that are worked on by all team members. Details can be seen in Fig. 1.

WP 1. Project Management (Saadat) keeps track of overall project progress and documentation tasks
WP 2. Infrastructure (Steffen) provides all needed infrastructure like server and software installations
WP 3. Data Sources (Kumushkan) takes care of selecting data sources and transformations
WP 4. Frontend (Christian) is responsible for the web frontend

Fig. 1 - GANNT chart

1
ARCHITECTURE

The infrastructure is built of four major parts as shown in Fig. 2. There are data sources on the web (HYDRIS,
OpenWeatherMap) which provide the input data. A processing server (Node-RED) transforms the data in regular
intervals and pushes it into a data storage. The data is stored in a 52 North SOS with a PostgreSQL database used
as storage. The website is directly accessing the SOS interface. A detailed instruction on how to setup the
infrastructure is provided in the projects wiki 1. The parts of the infrastructure are described in the following
chapters.

HYDRIS

SOS

Fig. 2 - architecture overview

DATA SOURCES

There are a lots of different data sources available on the internet. They are using a variety of different APIs and
methods to access as well as formats. We decided to use the water level data of the streams of Land Salzburg
HYDRIS(SALZBURG 2016) and weather data from OpenWeatherMap (OPENWEATHERMAP 2012).

HYDRIS is the hydrographic service of the province of Salzburg and is offering data on water cycle such as
precipitation, groundwater, water temperature and water levels of rivers and many more within Salzburg
province. We chose to query the water level data for several measurement stations along the rivers Salzach,
Enns, Lammer and Mur. The data can be downloaded as CSV file from the HYDRIS website.

1
https://gitlab.com/grmpfhmbl/16W_SDI_IP/wikis/installation-server
2
OpenWeatherMap is an online service that offers a variety of weather data (temperature, wind speed, wind
direction, precipitation etc.) worldwide. They also provide a JSON API free of charge. As finding stations in the
API is rather tedious, we choose to query the weather for the coordinates of the ZAMG weather stations in the
province of Salzbug2. The API will not necessarily return the weather for the same position, but for a coordinate
nearby. Thus the map shows more stations as there are ZAMG weather stations available.

SENSOR OBSERVATION SERVICE

52 North SOS (NORTH 2014) is an implementation of the OGC Sensor Observation Service standard (OGC 2012).
It is open source software under GPLv2. A SOS server stores data from sensors and offers them via a standardized
interface. If offers a variety of functions of which the most important are

GetCapabilities Provides information about the service, such as contact information, what type
of data is provided, what sensors are available etc.

DescribeSensor Provides a standardized sensor description

GetObservation Returns data for a provided sensor, phenomenon, time, or location

InsertSensor Creates a sensor inside the SOS server, so that data can be stored for it

InsertObservation Inserts data for a given sensor, time, and phenomenon

For every service we inserted one procedure. The InsertSensor parameters can be found in the project wiki 34

NODERED

OVERVIEW

Fig. 3 - Node-RED user interface

2
https://www.zamg.ac.at/cms/de/klima/messnetze/wetterstationen
3
https://gitlab.com/grmpfhmbl/16W_SDI_IP/wikis/hydris-insert-sensor-template
4
https://gitlab.com/grmpfhmbl/16W_SDI_IP/wikis/openweather-insert-sensor-template
3
Node-RED (IBM 2016) is a web based software product that allows to connect different APIs, data sources and
web services. It was developed by IBM and is open source under Apache 2.0 license. To create new applications
(called flows) one wires together different nodes (a functional block) that transform data from their inputs to
their outputs. Thus more or less complex flows can be created. The basic principle is, that a message moves
through the flow. A node can produce more than one output message for one input message. The user interface
can be seen in Fig. 3. In the main panel in the middle is the main working area, where flows are created. Ifferent
flows can be navigated to via the tabs. The left panel shows the available nodes. The right panel detailed
information to the selected node, or the debug panel.

There are many specialized nodes available (node-contrib-*) that can be installed over the web interface and
that add new functionality.

FLOW OPENWEATHERMAP

The flow for accessing the OpenWeatherMap API is shown in Fig. 4. A timer automatically runs the flow every 15
minutes. The first node stores a list of weather stations and their coordinates as downloaded from the ZAMG
website. This list is split into single lines and for every line a call to OpenWeatherMap is made. The
OpenWeatherMap is a custom installed node.

After the API returns the resulting JSON contains all measured values for the requested coordinate. This is split
up in multiple messages to create a single InserObservation call for every measured value. The gathered data
will be inserted in a prepared template and then a HTTP-POST node sends the call to the JSON endpoint of the
52North SOS. The limiter node ensures, that there is a maximum rate at which messages are send to the SOS
server. The green nodes are debug output nodes. Once activated they write information in the debug panel of
Node-RED. The complete code of the flow can be tested and seen in the Node-RED installation5

Fig. 4 OpenWeatherMap flow

5
http://104.199.84.35:1880/#flow/b7f86b0e.a80268
4
FLOW HYDRIS

The flow that extracts HYDRIS data and inserts it into the SOS is shown in Fig. 5. A timer triggers the insert every
morning at 4am. The CSV with the measurements of every station specified in the HYDRIS station data is
downloaded. Then the CSV is split up into its lines. Each line contains a measurement. After that again a template
is filled with the measured data and then send to the SOS server. The flow can be tested and seen in the web
interface6.

Fig. 5 - HYDRIS flow

FRONTEND WEBPAGE

The web page can be seen in Fig. 6. It is built around a map that shows all available stations. The menu allows to
choose between the HYDRIS dataset and the OpenWeatherMap dataset. The right panel shows the measured
data, as soon as a measurement station is clicked on.

For data visualization we used the Sensor Widgets created by Oscar Fonts7. These are widgets that allow to
connect to a SOS server and directly visualize the data. We used the following widgets which are shown in Fig. 7

Map This widget visualized features in the SOS on a map showing markers. It uses Leaflet
as mapping library. This widget is the centre point of our webpage. We slightly
customized it, as it could not emit events on selecting a marker.

Compass When selecting a marker on the map, a popup containing a compass is shown. This
compass widget shows the last measured wind direction.

Timechart The timechart is used to show temperature and humidity, as well as air pressure and
water level.

6
http://104.199.84.35:1880/#flow/21f6d7c5.5fbf78
7
https://github.com/oscarfonts/sensor-widgets
5
 Windrose The wind rose shows the history of wind direction and speed for a selected station. It
shows the data of the last seven days.

The frame is built with help of Twitter Bootstrap. A web GUI library available as open source under MIT license 8.
This library was used to create the menu bar and the accordion sidepanels.

Fig. 6 - web page

Fig. 7 - all used sensor widgets (compass, timechart, and windrose from left to right)

8
http://getbootstrap.com/
6
DISCUSSION AND OUTLOOK

This project shows, that it is possible with relatively few components, to create a complete infrastructure that
can collect data from various sources and offer it in a standard format.

Node-RED is a tool that is simple to use, also for people who dont have a lot of programming experience. Thus
after it is setup it is possible to let also inexperienced users with only little programming knowledge create their
own data conversions. Installation on Linux was straight forward and the installation description provided by the
developers worked well. Also there are a huge number of nodes already provided and can be found on the
internet, so that standard problems can be solved fast. There are dashboard widgets for Node-RED that seem
to offer useful functionality, but we did not explore that further. This should be done in the future.

The biggest problem with the web interface is, that despite it being able offer logins to different users it is not
really capable of being used by multiple users at the same time. When two users change even different flows at
the same time, changes will be overwritten. Also the logging and debugging capabilities are quite disappointing.
The debug panel can only show messages of a certain length and after that the message is cut off. For analysing
how data was transformed that is not really helpful. Also stepwise execution of flows would be helpful to have.

We used the JSON endpoint of the 52 North SOS Server. This endpoint is not standardized. So our webpage
possibly will not run with other SOS implementations. But as it mirrors the standard XML interfaces and is easier
to use from JavaScript, we chose to use it.

More measurement stations should be easy to add by just adding them to the input data nodes. The provided
exampled should make it easy to also add new data sources by creating new flows.

REFERENCES

IBM (2016), Node-RED website. Retrieved: 2017-02-28 Last, from https://nodered.org/.

NORTH (2014), 52North SOS. Retrieved: 2017-02-28 Last, from
http://52north.org/communities/sensorweb/sos/.
OGC (2012), OGC Sensor Observation Service Interface Standard, v2.0. SOS 2.0. STASCH, C. & ECHTERHOFF, J.
http://www.opengeospatial.org/standards/is, The Open Geospatial Consortium (OGC).
OPENWEATHERMAP (2012), OpenWeatherMap - website. Retrieved: 2017-02-28 Last, from
https://openweathermap.org/.
SALZBURG, L. (2016), HYDRIS ONLINE. Retrieved: 2017-02-28 Last, from
https://www.salzburg.gv.at/wasserwirtschaft/6-64-seen/hdweb/.