Home 4_Storing and processing data

4_Storing and processing data

Like every competition, also an energy consumption competition needs to store data in order to keep track of everything. As Figure 1 illustrates, the data is stored in a central point that is accessible by every competitor. Each participant could then retrieve the information it is interested in and interpret this for his own display. The initial idea was to use an existing data logging service. but due to some reasons that are described below, eventually an own database was built.

The first choice was to use emonCMS which is part of the OpenEnergyMonitoring project. It is able to log and visualize energy data in real-time. Version 3.0 is used in Appendix_ to demonstrate the connection of a home energy monitoring system with the internet. It uses JSON to send data from the arduino to the website. Unfortunately, it couldn’t be used for the competition because it isn’t able to send data back to the Nanode (or EmonBase as they call it). It was not in their interest to add this functionality so another solution was sought.

Second on the list was Pachube.com. It is widely used as an Internet of Things applications and has tutorials online for Arduino and Processing (http://community.pachube.com/tutorials) as well as code examples that could be used to connect Nanodes to the online service. This service uses curl as a protocol to update their feeds. A ‘POST’ function was successfully written to Post data from the sensor to the internet, see appendix_ . A similar looking ‘GET’ function would have been able to retrieve that data from the internet. But at the time of version 2, Pachube’s API did not support averages or medians over a certain period of time. Although this does not prevent to have enough information for a competition display, a lot of data would need to be transferred to make the calculations. In addition to this slow transfer of data, the calculations could be too complex for an Arduino due to the low amount of available memory.

Thingspeak is another online Internet of Things service and does about the same things as Pachube but does support averages. Being able to calculate with user-friendly averages reduces bandwidth and complexity and was one of the main reasons to change from service. Due to a previous project-experiment (appendix_), some experience was already present. The code used for Pachube only needed a small adjustment to work with Thingspeak. In appendix_ , the code of a version that is capable to send and receive values from Thingspeak is printed. Multiple values can be posted at once while only one value can be retrieved at a time. The connection between the Nanode and the server works, but is not really fluent. Sending and receiving multiple values per second fails. Using time intervals of at least 5 seconds turns out to be much more successful. This means that in a one minute interval, 1 upload and 10 downloads would be possible.

A PHP script was written to combine multiple values into one output and was uploaded to another server than the one of Thingspeak. It appeared that the Hub was not capable to access two different ip-addresses without rebooting. One way to trick this problem was to pass all data communication through one server that also forwards the data as Figure 3 illustrates. The ‘1 upload and 10 downloads’ can now be done within 10 seconds. Although this solution works, it isn’t optimal. It depends on two individual servers and if one fails, everything fails. The connection time increases a lot when using this intermediate step and sometimes connection timeout errors occur.

The cause is the fact that the data isn’t ready at the moment it is requested. The script needs to download multiple values from Thingspeak using separate http requests. One way to solve this is to anticipate. The intermediate server could store the data in advance and return it immediately when requested. The advantage of such a setup is that at the same time the interpretation of the data could take place on this powerful machine instead of on the Arduino. And if an own database is build up for this project, it could as well store the values of the competitors directly instead of making a detour to Thingspeak.

next: setting up a database

1 comment

Setting up a database | Stijn Coppens Labs Maart 1, 2012 at 3:20 pm

[…] of some reasons (discussed here), the decision was made to build an own database to store sensor data instead of using one at […]


Leave a Comment