Hi,
I have installed since 3 months an emonTx sending values to a raspberry pi on which emoncms is running.
The emonTx is sending every 5 seconds 4 values (3 phases power + 1 temp).
I have setup very simple feeds mainly a global power + kwh and kwhd.
So far it is running for 3 months giving more than 950'000 rows in the mysql database.
Everything went fine, however I started to look carefully at the results and some days, the results of the kwh and the kwhd where quite wrong. I started to analyse more precisely and I found that the emoncms on raspberry wasn't fast enough on some request to store the data.
I even discovered that some kwd values where not only increasing but sometimes decreasing indicating negative power. After a deeper look, I found the some request where stalling leading to unordered value storage. If this isn't an issue for power values, it is a real problem when computing kwh and kwhd.
So, most of my kwh values where simply wrong.
The evidence of my analysis comes from the http access logs:
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:00:09 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=621,112,1141,2000 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:00:01 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=624,112,1146,2000 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:00:14 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=624,112,1134,2000 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:00:27 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=623,117,1135,2000 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:00:34 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=625,115,1144,1993 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:00:40 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=573,113,1127,2000 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:00:54 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=574,111,1134,2000 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:00:47 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=571,112,1128,1993 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
127.0.1.1:80 127.0.0.1 - - [20/Mar/2013:00:01:00 +0100] "GET /emoncms/input/post?apikey=abc&node=11&csv=572,112,1136,2000 HTTP/1.1" 200 285 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
(Sadly, by default, the log is not showing the time to process the request)
and the database content:
Time Unix time Power Kwh Diff between measures
00:00:09 1363734009 1874 2930.020996 0.004150391
00:00:14 1363734014 1870 2930.026855 0.005859375
00:00:27 1363734027 1875 2930.036865 0.010009766
00:00:34 1363734034 1884 2930.043701 0.006835938
00:00:41 1363734041 1819 2930.043457 -0.000244141
00:00:47 1363734047 1819 2930.042969 -0.000488281
00:00:54 1363734054 1822 2930.046631 0.003662109
00:01:00 1363734060 1820 2930.046631 0
Hopefully, the base power values are correct, so I can run a sample stored procedure to fix the kwh and kwhd values.
One could argue to simply increase the elapse time between measures, but I don't want to loose precision and I even saw some request taking more than 20 seconds. It will probably go even worse as database size increases.
So now, I will really look forward to change hardware, as the raspberry will never be enough powerful for even simple processing.
Thanks for reading,
Re: emoncms + raspberrypi computation errors
Interesting, thanks for your post with debugging details. I have experienced this issue with certain SD Cards that dont seem to be as good as others. We had some Kingston SD cards here that I think had a really slow write speed so that all the input recording and processing was stacking up with the results that you describe. What is your raspberrypi system load?
Re: emoncms + raspberrypi computation errors
the results of the SD cards that we tested are in the wiki here http://wiki.openenergymonitor.org/index.php?title=Raspberry_Pi#SD_Card_Speed_.26_Benchmarking
Re: emoncms + raspberrypi computation errors
I have found recompiling histogram data to work fine on the pi, so recompiling the histogram and kwhd data from your exisiting power data is a viable option. I uploaded a module that converts a power feed to a histogram feed with grater detail than the current histogram processor here: https://github.com/emoncms/feedconvert
To run the conversion you just call:
http://ipaddress/emoncms/feedconvert/convert?power=POWERFEEDID&histogram...
Its a module thats in very early development, hence the lack of documentation
Re: emoncms + raspberrypi computation errors
The load of my raspberry is :
Linux 3.2.27+ (raspberry) 16/04/13 _armv6l_ (1 CPU)
22:07:49 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
22:07:49 all 6.69 0.00 8.95 62.10 0.09 0.04 0.00 0.00 22.12
I'm using a Transcend SD card of 8G class 10, I reach 16.4MB/s on writing.
Maybe 1 million rows on mysql on raspberry is too much for such a small system.