Can I post both signed integers and unsigned longs via the http post? - solved

In addition to being pretty new to emoncms I had too few hours of sleep last night, so maybe I am missing something obvious here... 

 

My setup is that I both want to send signed (i.e. data that is possibly negative) things like temperatures to emocms, but also large numbers, such as unsigned longs (such as sequence numbers coming out of the fairly advanced RS485 equipped energy meter that is measuring the energy consumption here). 

In the first case the system works out of the box, with the emonBase code building a signed, 2 byte integer out of the data received via RF from a remote Jeenode.

The code line in question in the NanodeRF_multinode sketch is line 178 (in my particular version of the code - should be about the same as the code in Git.)

int num = ((unsigned char)rf12_data[i+1] << 8 | (unsigned char)rf12_data[i]);

The above however causes problems (as far as I can tell) when I send in a 4 byte unsigned long value. If either of the two 2-byte parts of the 4-byte long value is above 32767, the number sent to the db (via the HTTP post) will be a negative number (due to how signed values are stored using 2-complements).

Right. I can compensate for that when processing the input, just before sending the data to the feed. It's pretty easy, just add 65536 to the input value. If the value stored is -10000, adding 65536 to this will give me the unsigned value I am looking for. Nice.
The problem is however if the value in the input stream is for example 5000. Adding 65536 to this will obviously give me an incorrect result (outside of the 16 bit range of an unsigned int). In other words: Adding 65536 only works if the value in the input stream is < 0, representing an unsigned value of > 32767. I could handle this by customizing the code, but that's what I want to get rid of by moving my home monitoring to a platform like emoncms.

Plan B then. 

I changed the above code line to read 

unisgned int num = ((unsigned char)rf12_data[i+1] << 8 | (unsigned char)rf12_data[i]);

Now I don't have to worry about adding 65536 when processing the input/creating the feed. Everything will be unsigned.
I however loose the ability to represent negative numbers, and given that I live in Sweden I am pretty sure the winter will bring temps way below zero..

 

To sum up: 

How do I properly handle both unsigned longs, and signed ints, without loosing precision in either case?

 

Thanks,

Göran

 

mountaindude's picture

Re: Can I post both signed integers and unsigned longs via the http post? - solved

Responding to my own question:

A new input processing action would fix the above, I believe. Something along the lines of "Signed to Unsigned", which would add a certain value (configurable?) to the input data, if the input data<0. Otherwise leave do nothing.

Hmm... Unless someone is already working on this I guess I'll have to brush up my php skills..

Robert Wall's picture

Re: Can I post both signed integers and unsigned longs via the http post? - solved

This is really Trystan Lea's area. I don't know the emonCMS system at all, however my feeling is that it doesn't actually matter now the data is passed around (as an array of char, signed integer or unsigned), it's how it is interpreted at the destination that matters. So I think you're right in thinking the problem is in the PHP.

As a work-around, can you add a number at source (i.e. base your temperature scale at -50°C, say) then subtract it again in the processing?

mountaindude's picture

Re: Can I post both signed integers and unsigned longs via the http post? - solved

So this actually turned to be pretty easy. Adding a new input processor to the already existing ones is next to trivial (if I've understood things correctly).

Making the following changes to the Models/process_model.php fully solved my problem:

After the existing process definitions:

$list[20] = array(
    _("Combine to long"),
    1,
    "combine_to_long",
    0,
    0
  );

 

The define the function combine_to_long at the end of the same file:

function combine_to_long($id, $time, $value)
{
  // Note: This process should ONLY be applied to the most significant word in a long
  if ($value < 0) {
    $value_msword = ($value + 65536) * 0.10 * 65536;
  }
  else {
    $value_msword = $value * 0.10 * 65536;
  }
  // Most significant word now ready. Add least significant word

 

  $result = db_query("SELECT value FROM input WHERE id = '$id'");
  $row = db_fetch_array($result);
  $value_lsword = $row['value'];

  if ($value_lsword < 0) {
    $value = ($value_lsword + 65536) * 0.10 + $value_msword;
  }
  else {
    $value = $value_lsword * 0.10 + $value_msword;
  }

 

  return $value;

}

 

A couple of notes:

a) I needed to scale down the values by a factor of 10. I took a shortcut and placed that scaling directly in the function. With a bit of retrospect I should instead have used the scaling process - it would make the above code more usable. 

b) The above input processor should be applied to the most significant word. For example, if a long value is sent from the tx node as node1_4 (least significant word = 2 bytes) and node1_5 (most significant word = 2 bytes), the above should processor should be applied to the node1_5 input, and node1_4 should be given as parameter to the processor. 

Works like a charm!

 

Trystan: feel free to include this in the standard code base if you find it useful. Or is the preferred way a git pull request?

 

 

TrystanLea's picture

Re: Can I post both signed integers and unsigned longs via the http post? - solved

Hello Goran, that's an interesting idea to move the data type conversion to emoncms as input processes, I wonder if there would be a better initial format for the nanoderf to send rather than 2byte integers?

Robert Wall's picture

Re: Can I post both signed integers and unsigned longs via the http post? - solved

You need a struct or a union in PHP!  SO has some suggestions.

mountaindude's picture

Re: Can I post both signed integers and unsigned longs via the http post? - solved

Well... I guess you just have to decide on a way to encode the data, and then stick to it. And then create whatever type casting functions that are needed. 

In my case I get 4 byte long integers via RS-485 from an energy meter. I get them as two two-byte words from the meter, LSW first. So at the tx node I just create a regular C long out of those two words, assign that long to a variable in an array and send it off over RX with the jeelib. Exactly the same way your tx node does, in other words. It's easy, clean and it works. I like..

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.