RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

I've been doing some work recently on a heating controller implementation building it from the ground up so as to get a better idea of the requirements: https://github.com/emoncms/development/blob/master/experimental/control/open_thermostat_scheduler/readme.md. I've also been looking at how emoncms might be built differently if the target is the raspberrypi connected directly to the rfm12pi board rather than a multi user cloud environment that is several steps away from the rfm network https://github.com/emoncms/emonview

I think a better integration between the nature of the rfm network that is predominantly used (structures etc), emonhub and emoncms or equivalent on the raspberrypi could provide a better overall system setup experience, how much information should the node transmit so that emoncms/view and auto describe the detected nodes? Controlling things also adds some new requirements and lots of questions in terms of control packet structures, acks and so on.

Then it would be good use the encryption that the rfm69 offers.

I've been trying to work through these ideas and questions and would be interested in hearing other peoples thoughts on what we should be thinking about, requirements and the best way forward.

items below:

- RFM Network encryption
- Automatic ID assignment
- Automatic node detection and description
- Polling powered nodes
- ACK's
- Control packets

RFM Network encryption
The RFM69 supports hardware 128bit AES encryption.
For encryption to work every node in the network must have the same encryption key which needs to be unique to that particular network. This raises the question of how to set the encryption key on each node when the system is being setup.  Sending the encryption key over the air as part of a pairing mechanism, even if that pairing window is short (I.e basestation has to be placed in a pairing mode manually for say 60s while the remote node is powered up), still makes it possible for a determined attacker to gain access if they have left a continuously listening node in proximity. It seems to me that the only way to do it securely is to set the encryption key as part of a wired manual setup procedure.  It could potentially be possible to do this by connecting the raspberrypi to the node and using an inotool + web interface setup (equivalent of using arduino ide but using the raspberrypi and the arduino ide abstracted through the raspberrypi web ui)

Automatic ID assignment
At the moment if you want to install multiple emontx or emonth nodes each node needs to be programmed with a unique nodeid. If your not familiar with arduino this can add quite a bit of complication to system setup. It could be possible to implement DHCP style auto id assignment so that nodeid's 1-30 are assigned sequentially to nodes as they are connected. If each node is powered up sequentially on node 0 it could start by sending a nodeid request command to the basestation. The basestation could then send back an allocated nodeid which could be stored in EEPROM. Technically this shouldn’t be too hard to implement but it does create a series of problems that may be harder to solve:

- Communication between wireless nodes currently relies on fixed nodeid's. Basestation would need ability to set nodeid settings on other nodes that listen directly adding another layer of development complexity, not impossible.

- Adds complexity to node and base station code that makes the code less readable, harder to understand for new coders and less likely to be modified, developed, extended in the spirit of hackable open source hardware.

- Setting the encryption key on the node may require direct programming anyway and so perhaps the setup complexity saved with auto id would be undone by the subsequent addition of encryption key setting.

- Another solution that might ease system setup without requiring an auto id solution would be a serial interface for setting the id and authentication key without needing a full reprogram, or a intool based raspberrypi programmer that could even be run from the web application.

Automatic node detection and description
It would be very nice if when you plug up a new node it appears in the list with its name, hardware version, firmware version, variable names, all automatically decoded values. There could be several different ways of doing this.

1) By nodeid
Nodeid's have been standardised for some time to refer to certain node types. Node descriptions could be fetched via a local or remote lookup table linked to the nodeid. Node 9 and 10 is usually an emontx, Node 19 is an EmonTH. The main problem with using nodeid's is that there could be many variations of emontx, different firmware versions with different packet structures and 30 nodeid's may not cover all the possible variations and then multiple installations of any of those variations. This problem could be solved using the same lookup table approach with a serial number packet:

2) By a serial number packet
Similar to nodeid the node could have a serial number which would allow a greater number of node types and version, that would then allow the basestation to perform a lookup either to a local lookup table or to an remote table that could hold up to date info. If the local table approach would be used a table update mechanism would be needed. This would either need a way for the basestation to ask for the serial number or to have an initial time when the node is powered up when the basestation needs to acknowledge that it received the serial number packet, likely to be less problematic than having the basestation able to poll at any time as that would require the node to be in listening mode but it would require resetting the nodes if the basestation lost its settings.

3) By an extended descriptor packet
The node could transmit all the information required for the basestation to decode and describe the node. As with serial number this would either need a way to make sure the basestation receives this information.

4) If the encryption key and nodeid was assigned when the node is connected to the basestation directly the descriptor for the node could be transferred from the node to the basestation at that point via serial.

Polling powered nodes
At the moment all nodes push their measurements at predefined intervals, say every 10s. There is no coordination of packet timing between nodes, this can result in packet loss if node tx times are aligned. Some nodes such as the emontx powered from the ACAC Adapter could have its readings polled by the basestation. The basestation could then sequentially poll all the powered listening nodes avoiding packetloss. Receiving the data back from the node is also an acknowledgement by its nature. Implementation would require a configuration in the basestation software to set poll times and gap between each node as well as timeout settings and then a script or thread within emonhub that generated the measurement request packets. A format and content to these packets would need to be decided upon and the code required to read these packets and send the measurements added to the nodes. Not impossible but would it offer a significant benefit? One nice feature could be the ability to poll a node faster for a shorter period, say power values at 1 second while looking at the data in realtime.

ACK's
When sending control commands it becomes more important to ensure that the control packet was received and so it may be useful to specify that an ack is required and have some way of relaying information about failed commands back to the UI  (could be implemented with a mqtt log topic and a filter for relevant messages in the UI). Some way of automatically retrying and setting the max number of attempts might also need to be added.

Control packets
In the heating controller concept I put together here https://github.com/emoncms/development/blob/master/experimental/control/...

The control node has a structure defined in the [nodes] section in the equivalent of emonhub.conf:

[nodes]

[[30]]
    nodename = thermostat
    names = state, setpoint
    codes = h,h

messages sent to the tx mqtt topic of the form

    tx/thermostat 1,1800

is then translated into a packet to be sent to the node for decoding at the other end using the struct approach used for sending node measurement data.

Should commands sent to particular nodes be limited in this way? There could be a second topic that will send encoded data directly.

Then there is the scenario where the control node also sends measurement data and needs a different decoder assigned to its node. Perhaps there needs to be two different definitions one for data going out to node 30 and the other for data received from node 30.

 

Interested to hear peoples thoughts

Trystan

haden's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Sounds very interesting.

I did have some similar thoughts about an automatic pairing of base and clients. My idea was to use either a cable or some sort of docking. It could be done with some pogo pins on the base station.

Another though was to implement the mesh network of the RFM69, but I don't know if the encryption is end-to-end or it need to decrypt/encrypt it at each node.

Regards

Kim

TrystanLea's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Thanks Kim, good to hear similar lines of thinking. Searching for mesh networking and RFM69 i came across this which looks interesting http://www.airspayce.com/mikem/arduino/RadioHead has anyone used this?

I've been looking at the control packet format item again, I think it would be good to allow control nodes to send data packets containing sensor data and receive a packet containing control parameters that have different payload structures.

At the moment following the emonhub config file approach decoders for receiving sensor data are defined like this:

[nodes]
    [[10]]
        nodename = livingroom
        names = temperature
        codes = h,

I've added in here the node and variable name properties. The corresponding structure on the sensor node looks like this:

    typedef struct {
        int temperature
    } Payload;
    Payload livingroom_sensor_values;

Lets say that livingroom node now has a relay attached to control a radiator actuator and that the control parameters that we need to set are: actuator on/off state and room setpoint, If it is sufficient to define a single type of control packet for a particular node that contains all the control parameters - which I think it is in probably most cases. We could have a second structure used for decoding packets received on the livingroom node

    typedef struct {
        byte state,
        int setpoint
    } PayloadCtrl;
    PayloadCtrl livingroom_control_params;

the emonhub config file would then need a separate section for describing packets that are outgoing, perhaps this would look like this:

[nodes_rx]
    [[10]]
        nodename = livingroom
        names = temperature
        codes = h,

[nodes_tx]
    [[10]]
        nodename = livingroom
        names = state, setpoint
        codes = B, h

When transmitting control data out from the basestation or any other node we could use the target node flag packet header as designed by jcw http://jeelabs.org/2011/01/14/nodes-addresses-and-interference/ or keep the header as the nodeid from which the packet was send and add another byte to the packet data itself including the target id.

interested to hear any preferences either way and other ideas for defining sensor node and control packets and whether to add the target id in the data part or follow jcw's recommendation of swapping the target with the sent from nodeid and then including the sent from in the packet data if needed

 

 

pb66's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Hi Trystan.

As you know I'm working on "control" in emonhub currently. I too think there should be separate Tx and Rx settings for each node, There maybe some instances where the payloads can be the same eg passing current meter readings to a Wh counting node but I think these would be few and far between, in the majority of cases they will differ.

Within the sketch the "Payload" data type created by the typedef struct is often given a name ending in Tx eg emonTx, I have for a while been using "PayloadTx" and "PayloadRx" as the types and that seems to work well.

In the emonhub.conf however I was thinking more along the lines of splitting the node into [[[Rx]]] and [[[Tx]]] rather than listing the node twice so that thier could be some common data eg node name etc and the complete node could be cut and pasted in (or even just appended to the file by a script) as one "node" 

[nodes]
      [[10]]
           nodename = livingroom
           [[[Rx]]]
                 names = temperature
                 datacodes = h,
          [[[Tx]]]
                names = state, setpoint
                codes = B, h

and in the sketches

typedef struct {
      int temperature
} PayloadTx;
PayloadTx emonTx; 

typedef struct {
      byte state,
      int setpoint
} PayloadRx;
PayloadRx emonRx;

I think we should definitely use jcw's "flagged as source or target", as then the devices sketch only has to deal with relevant packets rather than every sketch needing the ability to weed out the packets destined for that device. Especially as it makes sending the targeted packets so easy by effectively done by just adding 64 to the node id at serial.write() time. 

I don't think this is useable other than across the rfm network as it will impose a 64 node (id) limit which granted is double the current rfm limit but a restriction all the same. Yes we could add a different value outside of the rfm network but that may just be confusing and it's not totally "human readable" either as a sum has to be done (albeit a simple one) My thoughts were to expand on the flag idea alittle and use a negative number so node 10 is from the node and -10 is to the node, the minus sign being the source or target flag (so it works with bytes, ints and longs)

Although I like the potentially easy set up of DCHP nodes, the reality is even with my home network I use fixed IP's to make networking easier and I suspect the same may apply to the node network, combined with a need to swap out sketches, calibrate, pass encryption keys etc I think it is probably more secure, reliable and simpler to connect the node to a pc at some point.

However I believe that process could be made easier, maybe a custom UI to abstract the user from the IDE, git conf files etc. Certainly interlinking the sketches and datacodes/node settings would be a start, Robert and I thought adding the datacodes to the 3 or 4 non-int-only sketches could be a first step, so users can copy and paste from the sketch to the conf.

I have since  thought this through a little more and would like to experiment with easily recognizable __VARIABLES__ used in sketches, so that a bash script could be initiated to configure a node device connected by FTDI. Something like a " config_node 10 " command would look at emonhub.conf for node 10

[nodes]
      [[10]]
          nodename = livingroom
          device = emonTx v3
          sketch = some_generic_sketch.ino
          [[[Rx]]]
                names = temperature
                datacodes = h,
          [[[Tx]]]
                names = state, setpoint
                codes = B, h

And would replace the __VARIABLES__ in the sketch file with the values given node id, device frequency, group and of course an encryption key whilst it is in text format. Then compile and upload it to the node from the same command. so once the user has the details in the conf one command does it all with some reassuring status messages.

There are many mays something like this could evolve and I think a webpage where a user could select a sketch to suit thier needs from a short list for the chosen device, select a node id, specify rfm settings, choose variables and data types etc and this would provide "cut and paste" or downloadable data for sketches and emonhub.conf or for a "makefile" to compile. I previously commented on using something like the packetgen webpage to generate a datacodes string, this could be extended to provide a complete nodes section entry with matching typedef structs.

As you are aware I've been experimenting with I²C also, the reason I mention this is with regard to your "polling" comments. The way my I²C works is emonhub sends out a "broadcast" to trigger the calculations in each node, it then collects a frame each node sequentially, as long as all the nodes are polled between the broadcasts the data packets are all for the same exact period in time despite being collected at different times, the nodes also continue to collect data towards the next period whilst waiting to have the last periods totals collected. This works well as the previous periods totals are not zero'd until they get an ack confirming emonhub has them, so if not collected the new intervals totals are added to last so you never lose data.

This could also work with rfm although you have the added power conservation complications with battery powered nodes, but it is possible to ensure they are the first to be polled so they can sleep and a wake up timer set by the original "broadcast" would wake the node moments before the next "broadcast" effectively resetting the devices time counting every interval so it should never get out of sync.

 

 

TrystanLea's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Hello Paul,

great!

emonhub conf format

[nodes]
      [[10]]
           nodename = livingroom
           [[[Rx]]]
                 names = temperature
                 datacodes = h,
          [[[Tx]]]
                names = state, setpoint
                codes = B, h

looks great.

happy to for jcw's "flagged as source or target". I agree more and more about DHCP as I think about it, it just becomes over complicated and that the better approach is making configuration easier. Really like your idea :

"I have since  thought this through a little more and would like to experiment with easily recognizable __VARIABLES__ used in sketches, so that a bash script could be initiated to configure a node device connected by FTDI. Something like a " config_node 10 " command would look at emonhub.conf for node 10"

that would be a very neat way of connecting the two, solving the nodeid, decoding, descriptions, potentially encryption key all in one.

brilliant!

pb66's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

I have experimented a bit with using "easily recognizable __VARIABLES__ "  to do a "sketch merge" and found it is quite easily done. There are many questions around best way to implement it.

However, the (basic) method I currently have working is to use 2 markers disguised as comments so as not to effect the normal use of the sketch. The markers are added before and after the "mergeable" variables such as the node id and typedef structs etc.When the sketch is "merged" the 2 markers are replaced with block comment markers so the existing variables cannot be seen and the new variables are added. This seems to work ok and is very simple.

This method would mean any existing sketches can easily be made into "mergeable" templates by adding 2 comments and slightly rearranging the variable positioning in the sketch to group the "mergable" vars away from the ones to be retained. There also needs to be some info made available about each participating sketch eg what variables are available etc so the user can choose from them and configure their own payloads.

I think this will mean that there will be a need for some mapping between the sketches variable names and the users preferred input names. The alternative is too use individual variable markers so that the users names are merged into the sketch, but this may prove confusing when diagnosing a sketch issue. I think retaining the generic variable names in the sketch would be better when discussing the sketch and they are not seen after compiling so names are not important.

The "prototype" version I have also creates a "filecopy" of the sketch using date and nodename as a filename (eg 20150120_kitchenNode.ino) and/or saves the file in a temp location in the required "inotool" structure and calls inotool "make" and "upload" and/or creates a "Makefile" in the sketche's sub-directory for "aduino_mk" to use. (I have had both these "outputs" working but have not actually compiled yet as I'm working on a windows PC)

Paul

cagabi's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Hello,

I have been reading through this post and i am finding it incredibly interesting and have some things to add.

Encryption
I say this first because it is related to other points below.

I was talking the other day to Adam and Michael about the DHCP service and how to implement encryption without having to plug the node to the computer to write the key (which makes the DHCP pointless). I have been researching an idea that Adam had and ended reading about Public Key encryption, and i think this might be the answer. If you read carefully the introduction in here , you will understand how it works. A brief summary: you use the public key to encrypt and the private key to decrypt, if you try to decrypt with the public key what has been encrypted with the public key it doesn't work. That is why it is secure. This is called asymetric encryption while the one with only one key to do everything is called symetric encryption.

Sitiutation: Node A wants to send something encrypted to node B: A asks B for its public key (via a non encrypted message) -> A gets B's public key (via a non encrypted message) -> A encrypts the message with B's public key and sends it (this message can only be decrypted with B's private key that only B knows)

 

Automatic ID assingment
Well, i still think it is quite a good idea. For me the reason for having DHCP isn't to avoid pluging the node and uploading something, the reason is to avoid having to mess up with the nodes ids and keep a map (if you have plenty of them) to ensure you don't put the wrong node id when updating the firmware. We may still get into this problem if we want to use any kind of unique identifier but a serial number feels different that the node.

The main problem for me was the encryption, but using the Public Key would solve the issue. And something else that this encryption would solve is how the base sends the node id back to the node. An option would be to put the serial number in the same message with the node id, so that the node gets it. Another option is that the base broadcasts to all the nodes: "Registration successful, your node id is klsddjfdskfjhdslkjghdfskjlgh" (node id encrypted with node public key sent when requesting the registration). The nodes which are in the registration process will listen to the message, decrypt the node id and if they get an integer it is for them, or just in case another registering node gets an integer after decrypting (i guess quite difficult to happen), the message can be "node:3" explode it and get the integer. This applies specially to RF networks. Using IP adresses makes it easier.

 

Automatic node detection and description
I am not completely clear about the aim of this one. I think it is a good idea that the node sends and "extended descriptor package" after registering and keep an up to date table with this information but i can only see it for information purposes. This applies to a sensing unit.

 

Polling powered nodes
I think there is a good reason in your first post Trystan to think it is a good idea: "The basestation could sequentially poll all the powered listening nodes avoiding packetloss", That could be enough.

Another one that Adam was saying is that for sensors on batteries it is better to be able to request the measurement when you want it or to be able to set the intervals easily (Not in the node).

 

Control module trigered by an input processor
This was not in the original list but i want to ask. The approach i see in the  "Open heating controller/thermostat scheduler" is to have something running continuously, and (sorry i think in therms of emonCMS and not emonView) i guess in emonCMS it would be a module with a cron task.

But, what if you want to do some control as a reply to some meassured data? When talking with Adam and Michael they said and i agree the scheduled tasks seems easier, every now and again the module checks all the feeds on its list and if there is change on the value it does something.

I had a look and found out that to add input processors you need to hack the input module, bad thing. But it would be a great feature to have a processor that triggers the controller of a module that analyses the input and then sends some control signals o request some other meassurements if needed.

This would be great to combine with Polling data from the nodes. Your "control module" in emonCMS requests the data from the node (trigered by the scheduler or even manually), the node replies and the input triggers the control module that analyzes the meassurement and decides if something needs to be done.

 

Graphical user interface for setting up nodes
What a great idea, i love it

Cointrol packets
I see this is a hot thing. I have nothing to say but i will follow what you do and tell Adam and Michael they should keep an eye on this. This is very important as they are developing control units so it would be great if they can use what will be the standard way for emonCMS to set up and send control signals. You are doing an amazing work.

All the best 

   Carlos

ukmoose's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Looks good, but as per Twitter conversation of a few weeks ago.  It would be great to see compatability / integration with OpenTRV ( http://opentrv.org.uk ) built into the design.

 

The other thing is have you spoken to innovateuk ( https://interact.innovateuk.org ) they often have funding competitions which might be relevant.

TrystanLea's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Hello Carlos, great to hear your helping Adam and Michael at CAT, Adam mentioned you might be getting involved!

For people interested there is an ongoing project at CAT the center for alternative technology in Machynlleth Mid-Wales to develop monitoring and control technology for the renewables and heating systems on site, there was a blog post here http://openenergymonitor.blogspot.co.uk/2013/07/probably-best-student-pl... about the placement on which Michael from Aberystwyth University is on at the moment.

Thanks for sharing your thoughts on all this Carlos, Asymetric, public key encryption sounds like what we need, do you know if it would be possible to do this with the RFM69 using its built in AES encryption?

For the control module triggering, we where discussing here perhaps there is a need for being able to modularise input processing https://github.com/emoncms/emoncms/pull/280, perhaps we could implement some kind of hooks system that calls all module implementations of the hook function name when an event happens?

cagabi's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Hi,

Yesterday i spent a while doing a bit of research about the encryption. I  have started a new thread specific to this: http://openenergymonitor.org/emon/node/10164

AES is symmetric encryption, so it cannot do the work for us as we want. Anyway we have to keep it in mind as asymmetric encryption normally takes around 2 seconds to encrypt, that might be too much for our battery powered nodes. So maybe we can use the asymmetric encryption to retrieve the key to use with AES, but first we need to make asymmetric one work.

Cheers,

   Carlos

pb66's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Further to the idea of Auto-configuring node's sketches from the data held in in emonhub.conf, which I've had some success with, I would like to throw a related idea out there.

Basically it's to extend the "emonhub.conf" to use an additional "hardware.conf" and "firmware.conf", not unlike the "boards.txt" and "programmers.txt" used by avrdude and the arduino IDE.

These two files will be easily read/edited and hold profiles for "emon" hardware devices and firmware sketches. The files would only come into play when loading sketches and configuring hardware, again much like the avrdude files. They would not be needed for normal running and could be installed as a full library or just cut and paste in the profiles you need.

This would allow a unique device name and sketch name to be held in the emonhub.conf along with the custom node data to allow a automatic "plug and upload" system to be used to ease the task, eliminating (or at least reducing) errors to boot.

Paul

stuart's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

I'm currently looking at the above issues (in isolation so far), I currently have working AES128 encryption (done in software) along with a protocol for the sensors to describe their data loads/packets to emonCMS.

I have an aim to get the sensor data encrypted, sensors nodes to automatically be discovered/added/configured and fairly "plug and play"

Still in experimental stages at the moment though - and I also have the problem of the initial "seed" key being placed into each node.

 

This is mainly driven by the hundreds of IoT devices out in the wild which are completely insecure, and I'm fed up of reading about them :-(

 

PS: Also looking at how difficult it would be to do OTA updates from emonBase to each sensor

TrystanLea's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Hello Stuart, that sounds very interesting, Glyn and I met up with Damon from OpenTRV last week where he discussed what sounds like a similar approach. Keen to learn more about what your doing.

stuart's picture

Re: RFM Network dev next steps: Encryption, Auto id/info, ACK, Poll, Control

Sure, I've taken some inspiration from the "MySensors" project, although they use NRF24L01 radio chips

https://github.com/mysensors/Arduino/blob/master/libraries/MySensors/MyH...

I'll try and describe what I'm up to, but its still "playing about" at the moment!

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.