Emonhub dropping RX frame for 'non-numerical content'

Hi,

I've seen this issue brought up a few times for various reasons, but I am assuming that this is causing me to not be able to send data from my emonTx to my local emonCms. Can anyone offer any ideas as to why emonhub is discarding the received frames?

If this is indeed the same problem as others have reported ie. the timestamp is incorrect, where is it being reported from (emonTx or Jeelink) and how do it set the time? And whilst we are at it, what is the date format? I thought it might have been Julian but using an internet date calculator gave me a random date after conversion. I suppose this might be a function of the source date being incorrect.

It also looks like I have a 'blank' field after the OK but before the node id (10). If it matters, at this point I only have a single CT sensor plugged into the emonTX and it is being powered by the AC-AC adapter which is the base load (4w) with a test load of a 45w incandescent light which I turn on to see the value change (~50w). Is there a specification which lists what the values reported by the emonTx mean?

Thanks for your help,

Jason

Looking at the example log (found here: http://openenergymonitor.org/emon/sites/default/files/emonhub.log_.txt) and comments it appears that processing of an average (successful) frame should look like:

2014-08-17 09:42:18,471 DEBUG 3 NEW FRAME : 1408268538.47 OK 10 76 95 162 245 13 12 3 0 214 6 (-55) 2014-08-17 09:42:18,477 DEBUG 3 Timestamp : 1408268538.47 2014-08-17 09:42:18,480 DEBUG 3 Node : 10 2014-08-17 09:42:18,483 DEBUG 3 Values : [24396, -2654, 3085, 3, 1750] 2014-08-17 09:42:18,486 DEBUG 3 RSSI : -55 2014-08-17 09:42:18,523 DEBUG 3 Append to 'emonCMS' buffer => time: 1408268538.47, data: [10, 24396, -2654, 3085, 3, 1750, -55], ref: 3 2014-08-17 09:42:18,627 INFO Sending: http://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1408268538.47,10,24396,-2654,3085,3,1750,-55]]&sentat=1408268538
 

My emonhub log file

2014-12-02 22:11:47,982 DEBUG 3 NEW FRAME : 1417522307.98 > 210g
2014-12-02 22:11:47,982 WARNING 3 Discard RX frame 'information' : ['>', '210g']
2014-12-02 22:11:48,187 DEBUG 4 NEW FRAME : 1417522308.19  O i15 g210 @ 433 MHz
2014-12-02 22:11:48,187 WARNING 4 Discarded RX frame 'non-numerical content' : ['O', 'i15', 'g210', '@', '433', 'MHz']
2014-12-02 22:11:48,387 DEBUG 5 NEW FRAME : 1417522308.39 > 15i
2014-12-02 22:11:48,387 WARNING 5 Discard RX frame 'information' : ['>', '15i']
2014-12-02 22:11:48,592 DEBUG 6 NEW FRAME : 1417522308.59  O i15 g210 @ 433 MHz
2014-12-02 22:11:48,592 WARNING 6 Discarded RX frame 'non-numerical content' : ['O', 'i15', 'g210', '@', '433', 'MHz']
2014-12-02 22:11:51,027 DEBUG 7 NEW FRAME : 1417522311.03 OK  10 0 0 0 0 0 0 50 0 35 106 0 0
2014-12-02 22:11:51,027 WARNING 7 Discarded RX frame 'non-numerical content' : ['OK', '', '10', '0', '0', '0', '0', '0', '0', '50', '0', '35', '106', '0', '0']
2014-12-02 22:12:01,583 DEBUG 8 NEW FRAME : 1417522321.58 OK  10 0 0 0 0 0 0 50 0 123 106 0 0
2014-12-02 22:12:01,583 WARNING 8 Discarded RX frame 'non-numerical content' : ['OK', '', '10', '0', '0', '0', '0', '0', '0', '50', '0', '123', '106', '0', '0']
2014-12-02 22:12:12,142 DEBUG 9 NEW FRAME : 1417522332.14 OK  10 0 0 0 0 0 0 50 0 65 106 0 0
2014-12-02 22:12:12,142 WARNING 9 Discarded RX frame 'non-numerical content' : ['OK', '', '10', '0', '0', '0', '0', '0', '0', '50', '0', '65', '106', '0', '0']
2014-12-02 22:12:22,892 DEBUG 10 NEW FRAME : 1417522342.89 OK  10 0 0 0 0 0 0 49 0 192 105 0 0
2014-12-02 22:12:22,893 WARNING 10 Discarded RX frame 'non-numerical content' : ['OK', '', '10', '0', '0', '0', '0', '0', '0', '49', '0', '192', '105', '0', '0']
2014-12-02 22:12:33,451 DEBUG 11 NEW FRAME : 1417522353.45 OK  10 0 0 0 0 0 0 4 0 210 105 0 0
2014-12-02 22:12:33,451 WARNING 11 Discarded RX frame 'non-numerical content' : ['OK', '', '10', '0', '0', '0', '0', '0', '0', '4', '0', '210', '105', '0', '0']

pb66's picture

Re: Emonhub dropping RX frame for 'non-numerical content'

Hi Jason

The time stamp used is unix it is the number of seconds since 00:00 1st Jan 1970. The "1970 time issue" would manifest itself as an incorrect "1970" date in the leftmost column in the log file, so you are ok there.

The issue is indeed to do with the missing/blank variable or double space before the node id. From your previous post I assume you are using a JeeLink. Which firmware sketch are you using ? and which emonTx sketch are you using ? 

The variables passed depend on which sketch you are using, but the values you see are undecoded bytes so are not readable yet, emonhub is discarding the packets because the format doesn't comply with something it expects and can decode, because of the empty variable.

Paul

Jérôme's picture

Re: Emonhub dropping RX frame for 'non-numerical content'

Just as a sidenote, if we want to avoid the problem of consecutive whitespaces, we could change this line:

-frame = frame.strip().split(' ')
+frame = frame.strip()

See split definition here: https://docs.python.org/2/library/stdtypes.html

I don't think this would bring any unwanted side-effect. Otherwise, it is always possible to strip the result of split(' ') from empty strings anyway.

pb66's picture

Re: Emonhub dropping RX frame for 'non-numerical content'

I had considered something along those lines but couldn't decide which was better, the existing "fussy" code that can be a little annoyingly strict or the more "friendlier" method you suggest, the thought was there is currently no good reason for a double whitespace so as annoying as it is it maybe better than finding out you've dropped a value somewhere along the line and posted a load of rubbish to emonCMS because the byte values have been bumped along a position, so I left it as is for the time being,

Paul

jandkw99's picture

Re: Emonhub dropping RX frame for 'non-numerical content'

I'm using a brand new emonTx v3 from the OEM shop which arrived last week (assume  emonTxV3_RFM12B_Discrete Sampling sketch, but don't know). Unfortunately I don't know what the software version is and assume that I'd have to attach a direct interface cable to determine this. Is that correct? Am I able to determine this via the RF interface from the Jeelink?

Jeelink is similar, just delivered a week ago and using the default sketch [RF12demo.13]. 

Here is a sample of the output from the Jeelink user screen command. It is showing 'bad' frames, which I assume is relatively normal and they are just interpreted interference junk rather than actual data?: 

Current configuration:
 O i15 g210 @ 433 MHz
OK  10 0 0 0 0 0 0 0 0 46 112 0 0
   ? 210 10 12 0 0 0 0 0 0 0 0 70 112 0 0
OK  10 0 0 0 0 0 0 1 0 27 111 0 0
   ? 210 10 12 0 0 0 0 0 0 53 0 244 110 0 0
OK  10 0 0 0 0 0 0 53 0 119 110 0 0
   ? 210 10 8 0 0 0 0 0 0 53 0
   ? 210 42 12 0 0 0 0 0 0 53 0 79 110 0 0
OK  10 0 0 0 0 0 0 53 0 149 110 0 0
OK  10 0 0 0 0 0 0 53 0 151 110 0 0
OK  10 0 0 0 0 0 0 53 0 27 111 0 0
OK  10 0 0 0 0 0 0 54 0 138 111 0 0
   ? 210 26 76 0 0 8 4 0 0 114 0 24 112 0 0 26 81 128 4 2 128 0 0

 

Thanks again, Jason

jandkw99's picture

Re: Emonhub dropping RX frame for 'non-numerical content'

Hi Jerome,

Thanks for the reply.

I assumed that since the frame looked relatively complete that it was likely I could change the script to ignore the data, but I'd like to know what it is that I'm missing and why rather than just ignoring the space.

The other thing is that if I upgrade the software any change I make will likely have to be reintroduced.

I think I will make your change, just so I can see the monitor work with emoncms, but I really would like to know what the data I'm seeing from the emonTx actually means.

Thanks again, JW

Jérôme's picture

Re: Emonhub dropping RX frame for 'non-numerical content'

Yes if you modify your emonHub installation, the modif will be lost next upgrade. Consider this as just a test.

I was suggesting we could integrate this modification in emonHub, but Paul is right that the root cause is the double space and this is what should be investigated.

pb66's picture

Re: Emonhub dropping RX frame for 'non-numerical content'

Hi Jason.

The RF12demo.13 is still in development and changes are still being made to it over at JeeLabs, I did try loading it to a JeeLink I have a while back, but found a couple of anomalies and decided to fall back to RF12demo.12 until 13 is finalized and emonHub can then be altered for compatibility if needed.

It's too early to make permanent RF12demo.13 changes yet because then emonHub users, like yourself that have a working JeeLink could get stranded if JeeLib make a change and emonHub follows to suit without any backward compatibility, When it's finalised emonHub will almost definitely support RF12demo.13.

With that in mind it is also quite difficult to know with certainty the exact "vintage" of your RF12demo.13. If you want bleeding edge versions you could edit emonHub as suggested by Jérôme it certainly would help to test the theory and isn't that difficult to reintroduce to "updated" emonHub if needed, you could probably just add a hash (#) to what is currently line 95 of emonhub_interfacers.py so it comments out the latter half like this

frame = frame.strip()#.split(' ')

Alternatively you could upload the fully supported RF12demo.12 to your JeeLink until emonHub has support for RF12demo.13, I feel this may be a wiser choice as I'm unsure if any other differences will surface, for example I believe the RF12demo.13 also prefixes the firmware version with a new line when it prints that will cause an issue with the next version of emonHub (rc1.1)

Without the double space the data would be processed and you would see the resulting values in the logs if loglevel set to debug or info. The string you see, for examle " OK  10 0 0 0 0 0 0 54 0 138 111 0 0 " is node 10 after the initial OK and spaces, followed by 6 pairs of byte values eg "138 111", the trailing value (msb) multiplied by 256 is added to the leading value (lsb) to give you an integer and as they are signed ints, if the result is over 32768 subtract 65536 to give you a negative value.

The resulting 6 signed integers are power1, power2, power3, power4, Vrms and temp. Vrms and temp are both scaled by 100 so multiplying by 0.01 will give a value accurate to 2 decimal places, the rest are in watts.

Paul

pb66's picture

Re: Emonhub dropping RX frame for 'non-numerical content'

How did you get on with this?

Something else just occurred to me. emonHub rc1.0 was designed to suit just the RFM2Pi and that board comes in 2 flavors rf12 and rf69, the rf12 version uses a version of RF12demo that doesn't prefix with "OK" and the rf69 version does prefix with "OK" as well as providing a RSSI value in brackets at the end. emonHub rc1.0 won't recognize the use of "OK" without the trailing RSSI in brackets.

This has been changed in rc1.1 to also suit rf12 devices that use stock JeeLib RF12demo ie "OK" but no no RSSI. rc1.1 is "currently being tested" and if you wish to try it you can switch to the "testing" branch of emonHub.

​cd ~/emonhub
git checkout testing

should do it, but if you have made changes ie Jerome suggestion git may object so probably best to undo the change 1st.

Paul

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.