emonHub development - increase in scope of values over rfm network

As you may be aware there is a new version of OEM gateway called emonHub being developed and as part of that I have been exploring ways of receiving a wider range of data types and sizes from nodes over the RF network.

This method uses single letter "datacodes" either singularly to set a default type or in a chain to spell out the nodes data string format, much like the way we structure the data in an arduino sketch, a string of values held in a chain of types, (which until recently would of probably been all ints).

In theory there are 14 different data codes, but some are not supported by arduino and some are probably unnecessary (chart attached), in reality we could probably just use half of these.

This boosts the theoretical range to a potential minus 9.2 trillion'ish to plus 18.4 trillion'ish although testing beyond 14 digits of precision was not so precise, so +/- 100,000,000,000,000 (100 billion) is really the usable limit, in theory:-)

Please give it a try if can, it's very similar to OEM gateway in operation and the config file will be familiar but with a couple of additional fields. I've made a little script to install to pi with one line from the homedir.

git clone https://github.com/emonhub/dev-emonhub.git && dev-emonhub/install.sh

then the conf file needs to be edited, just add an api key for basic operation, then add node details to change data types.

nano /etc/emonhub/emonhub.conf

This is quite a complex subject so while I'm sort of expecting some formatting or calculation issues to surface once it's put to the test, but I'm also quietly confident, certainly with the more common ints and longs etc, not so sure about floats and doubles.

Attached is a screen dump of5 windows, the one on the left shows a part of the conf file, at the bottom of which you can see I have set a codes string of q, q, Q. From the bottom chart you can see that is 2 64bit signed Long Long's and a 64bit unsigned Long Long. Half way up on the left is a python script that posts a frame to socket 50011, that string is 25bytes long, (1 nodeid + 3x8bytes). When sent you can see the log tail in the bottom screen and the emonCMS input page at the top.

The second screen dump is exactly the same but a different string of 25bytes, showing the useful range within the 14 digit precision.

Although emonHub is not yet ready to release this is fully functional and any feedback from those of you willing to give it a test will help decide if it's useful. (please test fully before relying on it as it's very new)

The repo is at https://github.com/emonhub/emonhub/tree/development and you can visit https://docs.python.org/2/library/struct.html#format-characters for more info on the functions used.

pb66's picture

Re: emonHub development - increase in scope of values over rfm network

Putting aside the theoretical possibilities, Narrowing the choice to just 7 datacodes would match the data types readily available in arduino (see attached chart), it is apparently possible to use larger values in arduino but that's for another day.

Most sketches use code like this to "package" the outgoing data (this example is from emonTxV3_RFM12B_DiscreteSampling.ino)

typedef struct { int power1, power2, power3, power4, Vrms, temp; } PayloadTX;
      PayloadTX emontx;

in this package all the values are ints, this was standard and any software receiving the packet via an RFM network was designed to reconstruct any incoming "packages" as integers. To break the -32768 to 32,767 limits imposed by the ints, some later sketches use other data types, to my knowledge there are currently only 3 such sketches and they use longs to report watt hours. In these cases the data types are assigned for each value like this (example from emonTxV3_continuous_kwhtotals_noeeprom.ino)

typedef struct {
   unsigned long msgNumber;
   int realPower_CT1;
   int realPower_CT2;
   int realPower_CT3;
   int realPower_CT4;
   long wh_CT1;
   long wh_CT2;
   long wh_CT3;
   long wh_CT4;
   } Tx_struct;
Tx_struct tx_data;

These sketches break the int-only rules and are therefore not compatible with existing solutions, because the string of values can vary from sketch to sketch the recieving software needs to know that order to reconstruct the "package". And unless every node on a given network uses the same data structure, the receiving software must use the node id to identify which structure is being used

This data is given to emonHub in the form of a string of letters called datacodes for the example above, if that was for node 10 then we would have the following in the [nodes] section of the emonhub.conf file,

  codes = L, h, h, h, h, l, l, l, l

this tell emon hub any "packages" from node 10 are ordered unsigned long, int, int, int, int, long, long, long, long and emonhub can perform checks to confirm the recieved package is a valid size and reconstruct the values correctly.

The previous all-int example (call it node 11) could be specified using

  codes = h, h, h, h, h, h

which would provide an exact template for emonHub to use, so the size is set and can be checked or any node can have a "standard" datacode specified by using "code = " rather than "codes = " for example

  code = h

would tell emonhub that node 11 is all ints but doesn't specify exactly how many, so emonhub will just check the package size is a multiple of the given datacodes size.

OR in actual fact in this all-int example would still get correctly handled with no entry at all as emonhubs "RFM2Pi" listener (receiver) has also been set to ints to catch any un specified nodes using standard int-only sketches using a "defaultdatacode = " setting in the RRM2Pi settings.

emonhub always looks for the more specific data, so any node "codes = " trumps node "code = " trumps "defaultdatacode = " , the default emonhub.conf has "defaultdatacode = h" set so that by default emonhub works exactly like OEM gateway and to that you can add specific node data.

The  3 non all-int sketches and their corresponding "codes" are

emonTxV3_continuous_kwhtotals.ino                              h, h, h, h, h, l, l, l, l
emonTxV3_continuous_kwhtotals_eeprom.ino               h, h, h, h, h, l, l, l, l
emonTxV3_continuous_kwhtotals_noeeprom.ino           L, h, h, h, h, l, l, l, l

emonhub can be used in place of oem_gateway or the rfm2pi module or raspberrypi_run.php. It also runs on windows (with some settings changed) if you just  fancy seeing it in action, 


pb66's picture

Re: emonHub development - increase in scope of values over rfm network

In the interest of consistency I have changed the "code" and "codes" setting in the emonhub.conf to "datacode" and "datacodes" these terms are now used throughout the code and should really of been named like this from the outset. 

Updating this dev version emonhub via git without updating emonhub.conf will cause a problem that is easily remedied by just editing emonhub.conf by prefixing "code =" and/or  "codes =" with "data" eg

  datacodes = L, h, h, h, h, l, l, l, l


pb66's picture

Re: emonHub development - increase in scope of values over rfm network

As emonHub has evolved some details have changed and this although this thread may still be useful to those wanting to know more about using datacodes some details may be alittle misleading, so for clarity here's a bit of an update.

The "defaultdatacode" setting is now also just called datacode, the term "datacode" is now consistently used throughout emonhub's settings.

The "datacode = h" is no longer required in the settings file to make the rfm2pi decode legacy  "byte pair ints" as this is now hardcoded into emonhub's rfm2pi interface, it can however bee easily overridden by setting any single datacode in [[RFM2Pi]] [[[runtimesettings]]] or for a specific node using a single datacode or a string of datacodes in [nodes].

The accepted datacodes are currently limited to b, B, h, H, l, L and 0 (zero) just to simplify choice others can easily be 'switched' on if the need arises.

I hope this thread sheds a little light on what emonHub can do, without making it sound toooo complex. emonHub will function as a standard RFM2Pi interface by default, out of the box, without you even being aware this stuff exists. but if you need it it's there.


warlock's picture

Re: emonHub development - increase in scope of values over rfm network

What would be really awesome is to include the option to have the emon hub post to PVoutput as well,

there are simple php scripts out there, but they don't help when the internet is unavailable temporarily and don't buffer the data, Having hte option to enable data posting to PVOutput would be awesome.


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.