Emonhub buffering

I have Emonhub (version from Trystans Github) running on RPI sending to emoncms.org and local server on the same RPI.

I noticed at the weekend that my daily data on the local server was different to that on emoncms.org  When I investigated it seems I may have been having internet connection issues but the buffering didn't kick in to fill in the gaps.

I have the standard conf file settings of bufferMethod = memory and  bufferSize = 1000

I did some testing and when I disconnected my router the data was not saved to buffers on the Rpi.

Then, with the router back on, I changed the emoncms.org url to Xemoncms.org in the conf file and the data was saved to the buffer for over two hours before it was full and when the url was changed back it sent all the missing data to emoncms.org pretty quickly.

Why is Emonhub not recognising that the data is not getting through when the router goes down? Is there anything I can do to make it recognise this condition?

pb66's picture

Re: Emonhub buffering

That is very odd behavior, as far as I'm aware Trystan's versions have the same type of buffering that only deletes sent data if it receives a confirmation from the server, in most cases this is the word "ok", unless the response to a http request is "ok" it assumes failure and attempts to resend without deleting, or at least that's the theory..

Which version exactly are you using ? can you link to it or the guide you followed, as there are some variations?

Have you checked the emonhub.log file for clues? what logging level is set in the conf file?

Paul

nrgbod's picture

Re: Emonhub buffering

I just retested the buffering response to turning the router off and it's working ok now. I think I may have disconnected the Rpi by pulling the WiFi dongle earlier... the Rpi reboots when the dongle is re-inserted so obviously the buffers would be emptied, Duh!

I still have the original problem of why my emoncms.org Wh accumulators incremented more than they should have done at the weekend, but the local server accumulators matched my utility meters exactly.

The only thing I could find when examining the graphs where a couple of Jumps in the Wh data similar to the screenshots attached. Some of my feeds accumulated more than a few kWh extra over the course of the day on both Sat and Sun.

Any ideas what could cause this?

pb66's picture

Re: Emonhub buffering

Sorry, nothing specific really springs to mind, But since you have two sets of data you could down load csv data for the affected feeds for the period in question from both the emoncms's and compare then in excel to get an insight.

I can only suspect the wh accumulators have been upset by the data "flow" interruptions since both emoncms's would have been offered identical data and the only difference should be when that data was received since you are using wh totals. It would be easier to account for a reduced total through lost data, resetting pi or "rushing" the wh accumulators 25K filter, but too high a reading suggests double posting or something.

how many feeds are affected? Are the screenshots of the same feed at different scales? do have a graph from local emoncms for same feed and time?

pb66's picture

Re: Emonhub buffering

PS it may still be worth a look in the log to see when the requests were successful in relation to the data if it's not too late!

nrgbod's picture

Re: Emonhub buffering

The logs had been overwritten by the time I spotted the problem.

The first two screenshots where the same glitch at different zoom levels. I've attached a screenshot of the same period on the local feed which doesn't have the glitch.

There was 3 or 4 from 7 of my Wh accumulator feeds affected and also the same 3 or 4 from 7 of the Whd feeds.

nrgbod's picture

Re: Emonhub buffering

I have just checked the other feeds at the same point in time as the original screen shots and they also showed a step in the values at 16:40 although not of the same magnitude as the 1kWh+ step in the PV produced graph.

I wonder if this could be caused by an npt time adjustment or maybe the reply ok was sent but delayed getting back to the rpi because of say a DDNS ip address update so was sent again.

pb66's picture

Re: Emonhub buffering

I guess it's possible, although the effect seems quite extreme for a few second or, perhaps a couple of minutes overlap, which is what i would expect from a npt update. Without log data or knowledge of exactly what happened at that time it's hard to tell.

Hopefully if it happens again the log file can help.

nrgbod's picture

Re: Emonhub buffering

Thanks for the help Paul. 

I'll keep an eye on it and hopefully if it happens again I can catch the logs before they're overwritten.

Other than this little glitch emonhub has been running faultlessly for ages now.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.