Hi,
This may be a stupid question but here goes. I have a Pi running the latest oem_gateway (i.e. read only) with data being sent to emoncms.org. I have 8 inputs being sent to emoncms.org. I have a very slow internet connection (0.5MBit/s on a good day). How much data is being uploaded by the Pi to emoncms.org? Does the amount of data increase over time?
The reason I ask is for the last 24 hours my internet has stopped working for general use (slow and intermittent) which it normally does when downloading large files. So I switched off the Pi this morning so nothing gets sent to emoncms.org and within a few minutes my internet was back to normal so I am suspecting that eventually the data transferred to emoncms.org gets bigger as time goes by. Would this be correct?
Thanks.
Re: Data Sent to emoncms.org
That shouldnt be the case, but I did notice some strange behaviour on some of the accounts posting to emoncms this morning where very large bulk upload's where being sent. I havent had a chance yet to look into a solution, Im not sure if the latest version of the oem_gateway has bulk upload disabled. Jerome and I discussed disabling it until emoncms.org could better handle these kinds of uploads.
Re: Data Sent to emoncms.org
I reverted the changes introducing bulk send.
https://github.com/Jerome-github/oem_gateway/commits/master
Historically, the python gateway as introduced in the Rapsberry Pi module used bluk sending. When designing the standalone gateway, I moved to json to gain the possibility of having names inputs, thinking internet connections are reliable enough that bulk sending is not necessary. This assumption is probably wrong. I tried to reactivate bulk sending recently but had issues of another kind (I think the timestamp written in the database was wrong, perhaps not so hard to fix from emoncms side, but this wasn't the time for that).
I then stepped back to json mode.
The issue here, I assume, is that the data is stored in the buffer faster than it can be send. Therefore, quickly, you get a bottleneck, and data is sent 100% of the time. And at some point, you'll lose some data when the buffer is going to be full.
This is your bandwidth. The only option you have now, I'm afraid, is to lower the frequency of your measurements.
From a bandwidth point of view, bulk sending would be a solution, rather than a problem. Assuming it can be handled by emoncms. (Trystan raised the point of an emoncms.org downtime, after which all RPi would simultaneously bulk send the data they had stored.)
We can assume that users of the gateway either use version distributed in the ready-to-go SD card (without bulk), either do regular git updates and use the latest version (also without bulk). I doubt many people would be using bulk version, which was latest only between oct. 25th and nov. 13th.
Re: Data Sent to emoncms.org
Thanks Jerome for the clarification.
hendersoniain, do you know what your post rate is?
Re: Data Sent to emoncms.org
Hi,
Thanks for the info. I had set the emoncms.org post rate to 5 seconds (fastest possible). I will reduce this to say 30 seconds and try again. However, I am away from home for the next week and have left the Pi switched off whilst I'm away.
Iain.