Emoncms development log March 2014

Emoncms development March 2014

Thought I'd restart the monthly emoncms development log that we had a while back, several people have expressed interest in helping with development and that at the moment its either difficult to know what's being worked on and what the structure is for getting involved with development. The other two posts on emoncms development and wh calculation are more focused on particular features and the next release but I think from the discussions it would be good to have something with a bit wider a scope.

The idea with this post is that it is:

  • A list of all the emoncms developments currently being worked on,
  • Ideas for things to work on in the immediate future either for the next release or the release after
  • and longer term ideas so that we can all better see where this is going.

The following is a rough draft we can add to it as needed.

Next release v8

Emoncms v8, with new nodes interface, improved feed engines, csv export feature, optional use of redis and a start on a mobile friendly home energy monitor interface is almost ready for release. It now needs testing and documentation.

See the forum post here for more information on the v8 release: http://openenergymonitor.org/emon/node/3868
and the latest post for more detail on requirements for some of the v8 features: http://openenergymonitor.org/emon/node/3995

Emoncms v9
...

Features ideas and features being worked on

- Multi rate billing support: http://openenergymonitor.org/emon/node/2499
- Paul Reed, ukmoose, Improved versioning information, on a module by module basis in addition to overall emoncms version.
- GJPickard: Simple backup/restore option that can be accessed from the web interface.
- GJPickard: Syncing between local emoncms and remote emoncms.org to save needing to setup inputs, feeds, dashboards twice.
- Buffering on the raspberrypi when the internet is down, Jerome and I where discussing and working on this here: https://github.com/emoncms/emoncms/pull/118, fits into recent work on improving measurement reliability.
- Eric_AMANN: More flexible user authentication system, provide read rights to certain users, administrators for select users.
- batulzii: SSL/TLS
- Improved timezone support, locales rather than offset to allow for DST.
- An emoncms debian package, see thread: http://openenergymonitor.org/emon/node/3919

Control
- heating control scheduler, been working on this with Richard Hatfield, need to upload what we've done so far.
- near realtime message chain between browser and device being controlled and back again: https://github.com/emoncms/development/tree/master/experimental/control/...

Android App

- Aberystwyth university student project - need to upload work so far.

Ease of use

- lots of important work to do here
- automatic setup wizards..
 

Emoncms scalability
Not necessarily developments applicable for raspberrypi systems more for emoncms.org type deployments

- Investigate possibility of buffering feed writes in the feed engines so that writing can be done in blocks which is much faster than individual open, write, close operations. Coupled with:
- Feed storage sharding to multiple servers.
- Investigate in memory caching options for feed data
- In browser caching of graph data so that visualisations load like google maps in blocks.
- Event sourcing for rebuilding emoncms state in the event of failures.
 

- better test procedures, automated tests, continuous integration...

Development process

- Improving of emoncms development process, openness, transparency, structure, I hope this is part of doing that better.

This is an open thread, feature ideas are welcome. For detailed discussion on particular features it might be best to start new forum threads and link them to this development overview thread. I will try and repost the development summary every month with a short update on what happened last month.

Development meetup

Would there also be any interest in a uk or europe development meet up? Im happy to organise one up here at the OEM Lab in North Wales or we could choose a more central location depending on where people are. There a good space in Manchester or London that we could use or maybe we could attach something to the end of ElektroCamp in Leuven Belgium? which should be in April although details are yet to be confirmed for that.

Schism's picture

Re: Emoncms development log March 2014

This might get more of a response with the right year... :)

Thanks for this, very useful to help me understand what's going on in the near term. It's good to see that buffering on the Pi has already seen some work!

There are two things I have in mind to work on, I'm not sure when (if) they would be incorporated into a future version.

One is timezone support (using actual locales, like Europe/London, rather than UTC offset, to allow for DST); I haven't started to investigate this yet, but it would support a simpler implementation of my time-of-day filter process, which isn't quite ready for prime time since it doesn't work with DST.

The second is to investigate Debian packaging of emonCMS and other repositories to make the installation and upgrade process a little more intuitive (to me). This would be backwards compatible with existing installs, I just need to find time to do it (Debian packaging = not exciting).

Although this isn't strictly limited to emonCMS, I think it would be good for the shop to start selling an "emonGas" kit (call it what you like!) at its simplest an emonTH with choice of gas sensor (reed switch, hall sensor, optical pulse, or photoreflector) as a finished product. If this was to happen then the nodes interface from v8 would need to be expanded to cater for it.

I think a dev meetup would be cool, depending on interest (and enough notice). Edinburgh might suit most? :P

TrystanLea's picture

Re: Emoncms development log March 2014

Not sure how I missed that, long day yesterday I think :)

Good idea with debian packaging that would be very nice, I think it will be easier now that we dont need to compile timestore as part of the install process. I will add the points above

TrystanLea's picture

Re: Emoncms development log March 2014

We can add new node decoders as needed for new nodes, that shouldnt be a problem.

Jérôme's picture

Re: Emoncms development log March 2014

About the gas monitoring, we agree that there is a need.

The sensor seems to depend quite a lot on the meter to retrofit. The shop could offer a choice of several but which ones ? Besides, it could lead to frustration with people thinking that since it's in the shop, we ensure it will work. Before doing so, we may need a better view of which sensors work with each meter, and identify those that are more likely to work. I think I published here the results of my tests (one sensor only worked on my meter).

The software depends on the sensor (Reed switch or Hall effect), depends on whether the arduino is on battery.

The post-processing also depends on what we want to see. There's been discussion here about this (see link in gas mon page in "Modules"). Depending on what precision you have and want, you don't need the same calculations. I had to modify the processing in emoncms (changes on github) and I'm still unhappy with the results.

To sum it up, it is definitely something to be addressed, but it is far from obvious. (Isn't this exciting ?)

Not really related to emoncms, but I'd like to add the possibility to set the RF bitrate dynamically, like que freq, node and group. I'm currently stuck with the RFM2Pi-v2 sketch compilation. Not unrelated to gas monitoring, though, since the gas meter may be far from the base. In my house, I don't get the frames, hence the need to lower the bitrate. And while changing it, if I could make it a parameter, it would benefit to others. And it could take a new parameter in emoncms/RaspberryPi module.

Schism's picture

Re: Emoncms development log March 2014

Let me create an emonGas thread to prevent us going too far off topic..

I will commit to providing Debian packaging for v8 (assuming no unforseen obstacles) - I'll try to push a couple of branches this evening with this in place, and ask for test dummies. I really think this will make a few things easier (or at least, more transparent).

This topic is a welcome start, helping me understand current activities and where I can contribute. One further improvement I would suggest (under the "process..." heading) is some kind of structure in terms of where things are discussed so it's easy to follow up on interesting topics.

I don't necessarily want to recommend something specifically (i.e. GitHub issue vs forum topic vs both vs something else) but it would be useful to choose one way of working so that we know, for instance, that something isn't missed out because it was set up in the wrong place, and nobody else read it.

argofanatic's picture

Re: Emoncms development log March 2014

You bunch have done tremendous work here, hats off to everyone that has contributed. I'm hoping to be able to contribute some hardware and software that I have been working on since I found this site two months ago.

 

With regards to data volume, may I suggest filtering at the node. I have been monitoring 6 DS1820 sensors for two months now but my node will only post a temperature reading if the new value differs from the previous value by .25 degrees. The outdoor temperature varies considerably, but internal house temperatures seldom vary over the course of a day. All the graphing functions work fine, but the data storage requirements are minuscule. I plan on doing the same once my energy monitoring sensors get installed. 

 

Just my two cents (or pence).

 

Cheers.

TrystanLea's picture

Re: Emoncms development log March 2014

argofanatic: this is where the different feed engines for different requirements come's in, for your example you'd want to use a variable interval engine such as phptimeseries that stores a timestamp for each datapoint rather than the others as they rely on a fixed time interval between datapoints in order to achieve their query request time performance, which if the property measured is changing regularly is the best way to go.

That said even if what your monitoring doest change that much using a fixed interval engine has its advantages, there's no interpolation between datapoints implemented in the variable feed interval engine at present so if you wanted to compare your outside temperature to the internal temperature at a particular time - at the moment- it would be easier to record both with a fixed interval feed engine with both set to the same time interval.

I think its about evaluating what's the best solution for a particular application rather than one being better in all cases than another.

To make the filtering possible, that would be a matter of an input processor, that dropped the input update if the change is less than a given threshold or allowed it to go through to the log_to_feed process if it had changed. It would then need to automatically only allow the use of phptimeseries as the feed storage engine, all of which would be possible to do.

GJPickard's picture

Re: Emoncms development log March 2014

Am I too late for V8?

I have been messing around a bit more with count based feeds recently:  both pulse based counting where I need to do the cumulative counting (in emoncms at the moment); and where I am intercepting a cumulative number from another source.  When I get rogue readings or when packets are missed it would be good to be able to edit the feeds to sort this out.  Unless I have missed the function realtime edit seems to only allow individual corrections or a multiply by group correction.  Is it possible to include and an add/subtract option for a range of readings?

Also, any chance of a function to run processes on historic data?  What I am thinking of is when new processes are added or errors corrected then it would be good to be able to either generate brand new info from the historic data (e.g. min/max) or correct the bad figures.  At the moment, if you add a new process to an input it will only work with new data.

Many thanks

Gary P

 

pb66's picture

Re: Emoncms development log March 2014

Feature Request

Is it possible to make feed values available for input processing ?

Rather than using + x / by input could we have + x / by feed, this would not only make the processing more powerful but also more concise and probably more efficient. I appreciate this is probably not a simple task but I think it would be a huge advance in capability.

Paul

fluppie007's picture

Re: Emoncms development log March 2014

For gas monitoring check Flukso's way of doing it: https://www.flukso.net/content/gas-probe

charliemic's picture

Re: Emoncms development log March 2014

Thanks for the update guys, the future sounds bright! A meet-up sounds interesting, would be great to meet people and discuss ideas face-to-face.

Chris

Eric_AMANN's picture

Re: Emoncms development log March 2014

Hi,

I guess I'm late but it would be great to be able to configure EMonTX from EmonCMS.

See my related post here : http://openenergymonitor.org/emon/node/3623

Regards,

Eric

 

Robert Wall's picture

Re: Emoncms development log March 2014

"to be able to configure EMonTX from EmonCMS." would only be viable given a mains supply, and would at the least mean listening to the radio for messages. What changes to the configuration do you have in mind? Almost everything inside the emonTx depends in some way on the hardware and component tolerances which, excepting ageing effects, surely should never change.

There are also security implications. What happens if it suddenly starts reading half, or double, as a result of a message not intended for it - or a malicious message?

Eric_AMANN's picture

Re: Emoncms development log March 2014

Hi,

I have in mind two very different idea:

1. the ability to listen to the radio for messages only during the setup function to get some constants.

As an example, it may avoid modifying the sketch every time one need to change the node id. During the startup, the EmonTX sends a message to the EmonCMS with a default node id (let say 10) to say "hello, I'm a new node and I'm waiting for getting my node id". After validation from a logged user in EmonCMS, one could send to that EmonTX its "real" node id (any value except 0 and 10).

2. the ability to upload a sketch from EmonCMS to a particular node.

I understand the security implications in particular regarding the second idea.

Eric

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.