Archived Forum

OK. I have sorted out (1) myself; was missing the lines for network group in both

emonhub.conf-  line 55

and in the arduino sketch-   line 113

Hi,

Great to hear of your project. Sorry for slow reply. Here are my answers 

1. You need to have seperate network groups for each system that you want to keep sperate. E.g. all RF nodes on 210 (default) will be picked up by the emonPi. If you changed nodes to a different group e.g. 200 the emoPi on 210 would not be able to receive. Network group can be changed in emonTH / emonTx firmware and set emonPi group in emonhub.conf. 

2. Each node will have to have a different node ID. 

The emonPi node ID (default 5) is set here in emonPi firmware L122. You will also need to make corresponding change to emonhub.conf. See emonhub configuration info:

https://github.com/openenergymonitor/emonhub/blob/emon-pi/configuration.md

The ability to change the groups will help reduce the "noise" as neighboring groups will not be collecting packets destined for other emonpi's but it won't help with identifying the nodes at emoncms unfortunately as you are still restricted to 30 valid (1-30) node id's which won't cover the estimated 10 groups of 5 nodes (1 emonpi + 4 emonTx) 

Keeping each emonpi network as a separate group in this way is usually the better option and it's a neater way of doing things but if you do find that you have "borderline" devices that struggle to reach a designated emonpi group, a neighboring emonpi may offer a second chance if it's also within it's grasp and shares the same group id. 

There was a feature written into the "experimental" version of emonhub that was added for the purpose of extending and identifying the nodes from multiple groups, it is a setting called "nodeoffset" in normal running this would assume a value of 0 and it's presence go unnoticed, by setting this value to say 32 on a second device or interfacer it would give you a whole second range of node ids by simply adding 32 to each of the incoming packets node ids eg a second emonpi with no alteration beyond setting "nodeoffset = 32" in the conf would present itself as node 37 (5+32). setting a 3rd as "nodeoffset = 64" would be seen as node 69 etc.

I have tested this with a hand full of devices during development and although I see no reason it wouldn't work for 10 ranges I haven't tried it, 10 ranges of 32 node ids is probably far more than you will ever need but I chose 32 as an example as it makes the full range of node ids available for each device and prevents any overlap.

For example if you were to set a nodeoffset of 10 it will still work and it will give you a much smaller range of potential node ids (10x10) but it caps the node ids to 1-9 on each device as setting a node id of 12 would clash with the next hubs node 2+offset 10. and node 22 would clash with the 3rd node 2 etc.

Setting the offset to 32 is easier as it not only more robust, it also means no firmware changes (due to node id changes) since the RF networks remain as they are only the emonhub.conf needs changing. In addition to adding the setting you will also need to edit the [nodes] entries to suit the new range.

I have looked at the "emon-pi" variant of emonhub which was derived from that experimental version and the code seems to still be there but I do not know if it is still intact or functional as there have been many changes and this lesser known feature will not have been tested along the way.

If you or someone else would like to test briefly then editing the emonhub.conf to include "nodeoffset = 64" in the [interfacers] [[RFM2Pi]] [[[runtimesettings]]] and editing [[5]] to be [[69]] in the [nodes] section should work, if it doesn't appear at emoncms then take a look in the emonhub.log as it may well be the out bound interfacers that are preventing it working since they have had extensive work without awareness of this feature.

You mention the "cloud" emoncms, do you mean emoncms.org or your own instance? there is a 32 node restriction in emoncms, in a locally installed instance you can increase that restriction within the settings.conf and the "32" used here is a maximum count not a maximum id so if you do have 50 nodes across a range of 320 valid id's you can still set the max count to 50, 60 or 100 etc.

Unfortunately emoncms.org will only allow node id's < 32 so it is not really an option for 50 nodes.

Sorry if my explanation sounds complex but if the code is intact in the "emon-pi" variant then it is in fact very simple and quite probably the only way to squeeze 50 node ids out of the available 30, I hope it works for you.

Paul

Thanks for that.

Luckily we thought that imposing a large installation on emoncms.org would be unfair and set up our own cloud server- energy.waterford.sz.

The node offset will work just fine. I have tried an offset all the way up to 384, (32x12) and it works when I then set the node id from [[10]] to [[394]] for a Tx that has dip switch on 10.

Changing the [[5]] does not seem to do much, as I do not use the emonPi as a sensor.

So every emonTx box will show up uniquely on the cloud server. It does mean that the emonhub.conf will need to be set up carefully on each emonPi, but that will only be a once off.

Good to hear it is working.

With regard to the individual emonhub.conf's there are several options as emonhub was originally written offer the user flexibility and/or keep the configuration to a minimum rather than provide a pre-configured conf.

Firstly, as your node ids are all unique across your project you could use the same emonhub.conf copied to all 10 emonpi's, the [nodes] section could list all 40 active nodes and only the nodeoffset (and group?) need to be changed on each emonpi.

Secondly you could just delete the contents of the [nodes] section. The original code would run with no nodes defined if the values within those node's payloads were all integers which I believe the default emonTx v3 FW does. The downside of this very easy method is that the values would not get the "scale" corrections applied at the emonpi and would arrive at emoncms requiring scaling eg the voltage values would require a x0.01 process inserted at the beginning of the processlist.

there are other options in besides those 2, including for example this third option where you only define the essential data for the actual nodes used. so your [nodes] section would look something along the lines of (I have only included 2 emonTx's (9&10) per group here to keep it short) 

[[9]]
    [[[rx]]]
        scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1

[[10]]
    [[[rx]]]
        scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1

[[41]]
    [[[rx]]]
        scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1

[[42]]
    [[[rx]]]
        scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1

[[73]]
    [[[rx]]]
        scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1

[[74]]
    [[[rx]]]
        scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1

etc etc all the way up to your last active node id, this nodes section could then be used in all 10 emonpi's emonhub.conf. There is no need to define the emonpi or all the other unused nodes. This will only save you from defining the scales in emoncms so it may be tidier to just have an empty nodes section and do it all at emoncms?

There is something you should be aware of when using the emonpi with a remote server. The emonpi will forward the data in 30 sec batches and has no buffering, there is no confirmation of receipt before the data is deleted either. So if you have anything less than a perfectly reliable network connection and a server that is always able to receive and process data first time, all the time it will result in data loss as the emonpi's will simply "fire and forget" every 30secs. 

If you think this is likely to be an issue then for forwarding only, the original emonhub is by far a better solution not only for normal running reliability but also since you can set a larger buffer if required it facilitates server maintenance, my installs are all configured to buffer at least 24hrs data so a server can be taken down for maintenance etc and as long as it's back up within 24 hrs there will be no loss of data (providing there were no power interruptions to the Pi's as the buffer is in ram)

Paul

Things are looking better and better.

Yes, I did notice that the emoncms server already picked up voltage before a node was correctly defined, although out by a few powers of 10. I am mulling over the options that you suggest, although so far I think I will go for option 3, defining only the essential data for all the nodes in a common file. It looks neat and the further we move up the hierarchy, the less expert my operators will become. I work on the lowest level configuration and development, but I'm also training a technician in the IT dept to do the routine installation and maintenance Txs and Pis and that can include calibration. We would not like to have students risk electrocution, but we do hope to have a group of able and interested students enjoy creating feeds, building the dashboards and making apps as a community service project. It will make a better impression if the numbers are sensible as they come in to emoncms.

Data reliability can be an issue in Swaziland, where power cuts are frequent and Internet can suddenly slow to a crawl. We can live with an update every 5 minutes, which means that we can afford to lose a fair amount of data. Nevertheless it is worth looking at options for improved reliability.

1. How easy will it be to have an emonPi run emonhub software, as you suggest? I already have the emonPis.

2. We actually have two emoncms servers running: the cloud server, as mentioned, which I hope will soon be giving read access to the school website, but I have also set up a server on the school Intranet which has generator backup and is immune to the vagaries of the Internet connection. For now I have the emonPis feed into both of them independantly. However the best solution might be for the Intranet emoncms server to act as a buffer to the cloud server. Is there an easy-ish way to do that? There will not be a problem if the cloud server falls behind by a few hours, or even a day from time to time.

I'm not much of a mysql expert- I would need clear instructions- but running rsync as a chron job every few minutes on a set of files would be fairly straightforward for me.

Thanks for all the help so far.

Quinton

It could be fairly easy to get original emonhub (v1.2) running on an emonpi as it is at the heart just a regular raspberry pi, there is no official route or guide as the only current option promoted by OEM is the single user local emoncms SDcard which uses the emon-pi variant of emonhub.

As a forwarding only solution installing Raspbian, configuring the filesystem to be read-only and then installing emonhub v1.2 using the installer is pretty much all that is required for a very robust "fit and forget" install (each of these parts is documented in the main emoncms install docs). But that doesn't utilize the emonpi's  LCD and button, nor does it fully utilize the emonpi add-on board since the GPIO will not get configured for the OOK and/or the reset/power button etc, the emonpi board will however continue to function as both an RF transceiver and a 2CT energy monitor via the serial port (even though the latter part may not be important to you). Using the more "emonPi" type elements would require some dissection of the emonPi repo as they are not available separately.

Power outages at the emonTx's isn't normally a concern as if the powers out there is nothing to monitor, the emonpi's might benefit from adding a small battery pack perhaps? that would prevent minor power glitches from disturbing things and it is worth noting the current emonpi SDcard image has a writable partition and does do some persisting other than feed data, if data is being written when the power fails that would cause the whole SDcard to fail rather than just the write partition.

the original emonhub does not have a write partition and does not write anything to disc in normal operation so sd card failure is reduced significantly, although it could still benefit from a battery pack to avoid losing any yet undelivered data still held in ram.

The buffers are per target server so when you have multiple servers it would retain independent "queues" for each and deliver when it could.

It is worth noting that the "emonpi update scripts" will not be available for use if not using the full emonpi install and if you are currently using the emoncms "emonhub" webpage via a browser to edit emonhub.conf you would need to switch to using ssh unless the full apache/emoncms/sql stack is installed IMO thats a lot of complication for a potentially one of set-up, that way the firewall can be closed (except for ssh) and a secure key used for ssh access rather than 10 different (the same ?) user/password logins.  

If you are wanting to let the students loose on the feeds and dashboards you can have more targets set up in emonhub so that you can easily have accounts for different student groups and/or a master account too. Since you have access to the feed files and the SQL tables you can effectively "reset" accounts by deleting the user account, creating a new one, copying in the data and editing the apikey to the previously used one for the deleted account in the SQL table.

I do not know of a way to make one emoncms forward data to another and think you may stray into issues using a live rsync solution as some data is held in redis or read from the sql tables and not always read from the persisted feed files. it might be possible but at best it's unchartered waters to my knowledge.

Paul

I have some queries about your instruction.

1. Making a read only filesystem seems to assume that there is a /home/pi/emonpi folder, but after installing raspbian pi's home is empty. Do I install emonhub first?

2. Do install emoncms at all?

3. If I don't install emoncms, do I need to configure wireless using ifupdown and the /etc/network/interfaces file

My "instruction" was a very loose main points only generalization, We can work through the various steps if you are wanting to give it a whirl.

See the "Configuring Raspbian as Read-Only" thread for the original read-only instructions, they have been altered a fair bit for the emonSD image for the 3rd "data" partition which you do not need and rather than writing/editing the fstab that guide does seem to symlink a replacement which will only work if using a emonpi image.

There are additional RAM partitions for openHAB and dchp in the newer guide, if you are not using openHAB that partition (and directory) is not required. As for dchp I use a different method which works really well for me but I have no idea how it will mesh with any other "emonpi" networking strategies if you use adopt them.

I like to keep it simple and just symlink the 2 dchp files to /tmp.

    sudo mv /etc/resolv.conf /tmp
    sudo ln -s /tmp/resolv.conf /etc/resolv.conf
    sudo mv /var/cache/unbound/resolvconf_resolvers.conf /tmp
    sudo ln -s /tmp/resolvconf_resolvers.conf /var/cache/unbound/resolvconf_resolvers.conf

Raspbian wheezy had a bit of a problem with keeping a wifi connection alive so a reconnection script was used, I would say try Jessie without at first (if using wifi of course) and if you have issues you can choose a wifi script if required. 

The installation of ntp-backup is essential if you have anything less than a 100% reliable network and just to start off on the right foot ensure you "first boot" with a known good internet connection or run in "write" mode until the time is corrected, after that the time keeping will be at worst "in the right ball park" see the "Read-only time issues" thread if you want more details on how that works.

If you install ufw (uncomplicated firewall) be sure to allow SSH (port 22) before enabling it (unless you have a keyboard and screen handy).

Emonhub is a one liner install but you can also also add the menu commands to make it's use easier with

    wget -O ~/hub_utils.sh https://openenergymonitor.org/emon/sites/default/files/emonhub_utilities_0.txt
    chmod +x ~/hub_utils.sh && ~/hub_utils.sh

accessing the emonhub.conf is then as easy as emonhub -c etc, type emonhub -h for help.

It would be a good idea to setup logrotate to prevent the "/var/log" partition filling up with messages etc.

EmonCMS is not required for a forwarding only set up.

The interfaces file shouldn't need editing unless you plan to use fixed IP's or a non-standard network set up. If your network will recognize hostnames then just use the "change hostname" option after "sudo raspi-config" and name each emonpi something unique, otherwise you will need to allocate fixed IP's and depending on your network you then may need to add the details to the interfaces file, but that is usually only to setup inbound access to the pi from outside the LAN. 

Change the emonpi timezones to get the error logs to log in your local time, the data is actually passed with unix timestamps so it's not mandatory.

If just using wifi then you will need to add your ssid and password to the wpa_suppliment.conf file as usual.

I've rattled this off from memory so please forgive me if I've missed something out but the basics are covered I think. There is other stuff you could install like avrdude and the autoreset script if you plan on updating the emonpi FW remotely.

Paul

Oh and I just recalled the "easily overlooked" but crucial editing of the /boot/cmdline.txt to free up the gpio serial port. Since the launch of the Pi3 Ithink the bit to remove will include something like "console=/dev/serial0,115200" it's pretty well documented thoughout the site and guides but the device has recently changed from /dev/ttyAMA0 to /dev/serial0 or something pretty similar.

Hi Paul.

I may well give it a go. Before I start though, is there a reasonable way to monitor via ssh the inputs that emonhub is receiving and posting if I haven't installed emoncms?

Just if anyone is interested, the large node offsets only work well with my cloud emoncms, which is version 9.5. My intranet server (V8.5) does not , and the emonpi boxes, also v8.5, show green time stamps, but the text next to them is "inactive" and no data is displayed.  Maybe there is just a tweak to be made to the emoncms?

Is there any working image of the emonpi with emoncms v9?

emonhub log:

2016-05-05 10:48:08,655 DEBUG RFM2Pi 1439 NEW FRAME : OK 10 0 0 0 0 0 0 0 0 135 99 184 11 184 11 184 11 184 11 184 11 184 11 1 0 (-55) 2016-05-05 10:48:08,659 DEBUG RFM2Pi 1439 Timestamp : 1462445288.66 2016-05-05 10:48:08,660 DEBUG RFM2Pi 1439 From Node : 10 2016-05-05 10:48:08,660 DEBUG RFM2Pi 1439 Values : [0, 0, 0, 0, 232.39411187399998, 300, 300, 300, 300, 300, 300, 1] 2016-05-05 10:48:08,661 DEBUG RFM2Pi 1439 RSSI : -55 2016-05-05 10:48:08,662 INFO RFM2Pi Publishing: emonhub/rx/10/values 0,0,0,0,232.394111874,300,300,300,300,300,300,1 2016-05-05 10:48:08,665 DEBUG RFM2Pi 1439 adding frame to buffer => [1462445288, 10, 0, 0, 0, 0, 232.39411187399998, 300, 300, 300, 300, 300, 300, 1, -55] 2016-05-05 10:48:08,666 DEBUG RFM2Pi 1439 adding frame to buffer => [1462445288, 10, 0, 0, 0, 0, 232.39411187399998, 300, 300, 300, 300, 300, 300, 1, -55] 2016-05-05 10:48:08,667 DEBUG RFM2Pi 1439 Sent to channel' : ToEmonCMS 2016-05-05 10:48:11,181 DEBUG RFM2Pi Discarding RX frame 'unreliable content'? 167 168 103 207 58 58 244 181 182 22 184 143 210 150 213 26 78 104 135 120 230 (-100) 2016-05-05 10:48:11,688 DEBUG RFM2Pi Discarding RX frame 'unreliable content'? 3 150 250 172 18 159 131 179 214 117 225 153 82 115 250 187 58 204 128 249 218 (-101) 2016-05-05 10:48:12,833 INFO energywaterfordsz sending: http://energy.waterford.sz/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1462445268,10,0,0,0,0,231.73739944199997,300,300,300,300,300,300,1,-54],[1462445278,10,0,0,0,0,231.664431394,300,300,300,300,300,300,1,-53],[1462445288,10,0,0,0,0,232.39411187399998,300,300,300,300,300,300,1,-55]]&sentat=1462445292 2016-05-05 10:48:14,751 DEBUG RFM2Pi Discarding RX frame 'unreliable content'? 12 215 223 14 50 181 13 152 5 58 226 159 115 5 130 112 191 110 142 112 148 (-98)

emoncmslog:

2016-5-5 10:49:51 MQTT INFO Reloading config 2016-5-5 10:49:57 MQTT INFO Reloading config 2016-5-5 10:49:57 MQTT INFO Received mqtt message: emonhub/rx/10/values 0,0,0,0,231.928940568,300,300,300,300,300,300,1 2016-5-5 10:50:03 MQTT INFO Reloading config 2016-5-5 10:50:07 MQTT INFO Received mqtt message: emonhub/rx/10/values 0,0,0,0,231.90157755,300,300,300,300,300,300,1 2016-5-5 10:50:09 MQTT INFO Reloading config 2016-5-5 10:50:15 MQTT INFO Reloading config 2016-5-5 10:50:15 FEEDWRITER INFO PHPTimeSeries bytes written: 0 2016-5-5 10:50:15 FEEDWRITER INFO PHPFina bytes written: 0 2016-5-5 10:50:17 MQTT INFO Received mqtt message: emonhub/rx/10/values 0,0,0,0,231.892456544,300,300,300,300,300,300,1 2016-5-5 10:50:21 MQTT INFO Reloading config 2016-5-5 10:50:27 MQTT INFO Reloading config 2016-5-5 10:50:27 MQTT INFO Received mqtt message: emonhub/rx/10/values 0,0,0,0,231.965424592,300,300,300,300,300,300,1 2016-5-5 10:50:33 MQTT INFO Reloading config 2016-5-5 10:50:37 MQTT INFO Received mqtt message: emonhub/rx/10/values 0,0,0,0,231.928940568,300,300,300,300,300,300,1 2016-5-5 10:50:39 MQTT INFO Reloading config 2016-5-5 10:50:45 MQTT INFO Reloading config 2016-5-5 10:50:47 MQTT INFO Received mqtt message: emonhub/rx/10/values 0,0,0,0,231.755641454,300,300,300,300,300,300,1

This thread has is now continued on the new forum