Archived Forum

Hello Jerome, yes Glyn and I should have been a bit more communicative with you about our plans. When we created the first image we just did a git clone straight from https://github.com/Jerome-github/oem_gateway, we then tested its normal posting operation but we didnt really test the buffering extensively.

Glyn has recently made another image that fixes the ram issue with the new raspberrypi's. I will find out what commit version the current gateway sd cards are upto. Either one of us will test it before it goes out on SD cards and I wouldn't worry about breaking it at anytime and fully understand that the stability will change all the time.

I want to help with getting the bulk uploads to work so Il try and get involved with your thread.

Awesome guys, I'd love to help out, unfortunately I can't code so I am more than willing to test.

Hi Jerome,

Thanks a lot for your good work making the gateway, it does work very well. D

The new / current gateway image (22Oct) uses the older version of the gateway before the fix you mention. Users can update their gateways by running 

$ ipe-rw

$ cd oem_gateway

$ cd git pull 

Have you got some documenting deplaning how the  bulk send works? We would be keen to get a blog post up with info together with upgrade instructions. We we would love this to come direct from you. Would you like to write a guest blog post? This would ensure you receive full credit for your work. 

We will make sure the new version containing the fix is included in the new ready-to-go SD card image.

id you notice our chance of putting oemgateway.conf in the FAT / boot partition of the SD card to allow windows users to easily edit. Taking this a step further to maker setup even easier it would be great to have a simple web GUI running on a lightweight web-server on the Pi to enable users to configure the gateway settings from their web browser. I wonder it it would be possible to run a web-server from a read-only file system? Mmm I'll look into this. 

Keep up the good work. 

Hi.

My point is that software is never finished, so users may want or need to update the software on the Pi. This means regular

ipe-rw

aptitude full-upgrade

ipe-ro

but also updates of OEM software (until it is packaged in the distribution...)

It's nice that you used git so that the user just has to git pull. But this means that each time the latest master is broken (like it is right now), the possibility exists that users will pull latest changes and break their installation. This advocates for dev branches.

Another issue could be configuration modification. I may add parameters anytime, or change filenames, or anything. (I recently renamed the main script by removing trailing .py, which breaks automatic startup if the init file is not changed accordingly)

Regarding the bulk send, I don't see what there could be to write about. From the user's point of view, it is transparent, except for a newly added parameter : sending period in seconds. The only issue is that is does not work for the moment... Or perhaps I don't understand what you're talking about. The idea was to avoid a call to the server every few seconds, as it sometimes takes time to answer.

Just to make sure everything is clear, the commit I'm referring to in my first post is not a fix but a breaker, so please don't include it until it is fixed.

I already thought a config GUI could be neat. It could also be a multiplatform software (Qt ? command line ?) interacting with the Pi. Or any web assistant running on emoncms.org, providing a content to copy on the SD card before using it in the Pi. Anyway, the configuration should be self-explanatory. If it is not, then this needs to be improved. Then, a few examples could do the trick, and the rest is fine-tuning that could be done easily in the text file. i didn't dig any further. Seemed too much trouble for the benefit.

Note that the config file format may have to be changed if moving to Python 3 some day, as the library I used is an external lib that makes the usage of the config file quite practical, but it has not been ported to Python 3, and it's kind of orphaned. See here: https://github.com/Jerome-github/oem_gateway/issues/7

Hey Jerome,

With the good work you have done so far and hopefully in the future I think the Ready-To-Go-SD card is here to stay and why shouldn't it be, with regards to updates the to code, well anyone who is using it and updating it from github should have the insight to read the README, and this file should contain any important information about the update that could potentially be a problem like the init script.

As for a dev branch, well if it's not to much hassle then probably that would be a good idea too.

Never the less good work is being done here.

Apparently, the default value for the emonGLCD "sendtimeinterval" parameter is not 0.

This leads to issues with the RFM2Pi rev1.

I think this should be set to 0 by default.

I can add a note in the config file to suggest RFM2Pi rev2 owners that use an emonGLCD to set it to something else.

Perhaps would it be nice to publish a repo with the modifications you made for the card, so that we'd see what modifications you made and more importantly, we'd have a bug tracker for it.

Or we could use OEM Gateway's git repo for the bugtracking. Or a dummy repo on emoncms' Github account, with a few textfiles describing the content of the SD Card and the modifications.

Good point about GLCD sendtimeinterval being set to a default of 0, will change that in the ready-to-go image.

I've added a note about the rfm12pi v1 to the ready to go image guide here: http://emoncms.org/site/docs/raspberrypigateway

This rock solid gateway is really good, having experienced sd card failure.

now after starting to run rock solid gateway it looks like all the temperature nodes (2x funky sensors + emonglcd temp etc.) which are a bit further away drop out. These usually also update after a minute or so unlike other sensors which update several times in a minute. Due to 4 other nodes with some 3-7 sensors on each these less frequently sending sensors do have collisions and update at around 2minute intervals.

Is there a parameter which would extend the "keep alive" time or enable a regular search of sensors  ?

I don't understand your problem. To avoid collisions, maybe you can use reporting periods that are "prime".

I don't know what we can do from the emonBase to avoid collisions.

What mechanism are you refering to when you mention "keep alive time" or "search of sensors" ?

I have some 20+ sensors, those seem to drop out one by one starting from those which are least frequently used.

Did not have this problem with previous gateway firmware, the hardware is the same now.

Now rebooting (cron) once an hour to keep all sensors transmitting data.

My guess there is somekind of timer to drop sensors (keepalive) or alternatively there is a memory leak (??) causing the drop of sensors. Collisions is not a problem as such, it is this drop of sensor, particularly those which are less frequently updated and which are also more remote ones (physically).

hope this clarifies a bit whats the symptom..

OK my guess is with so many sensors, the gateway buffers data from the sensors faster than it can send it to the server.

You could activate the logs to check that.

This could be improved with bulk sending (ability to send several datasets in the same URL post) but this won't be implemented tomorrow.

In the meantime, perhaps can you try to lower the frequency for some sensors and see if it gets better.

There is no keep alive tome or anything, and sensors should never get lost.

Again, if you activate the logs (needs read-write mode, see help about this on the ready-made SD card help page), they will tell you a lot about what happens.

Makes sense - how is the buffer organized fifo or round robin ?, i.e. Based on what would it drop data ?

i need some pointers on how to activate logging as only installed the sd-card image and have not really looked into contents - unless you mean logging the output of the program itself (>/...).

Currently, the buffering feature is minimalist. It is FIFO. It could be improved. I think it makes sense to dimension it so as not to trash any data when a reasonably long outage occurs. Current setting is probably conservative, to avoid memory consumption. A lot here is still to be defined.

For the logging, I'm referring to this page: http://emoncms.org/site/docs/raspberrypigateway

Havent had time to do logging for the issue described above, but found a new "behaviour".

The emonGlcd seems to be hanging regularly after putting rock solid gateway in use - thish hang up looks like happening regularly at 0:00 hours. Could it he possible that the time message to emonglcd is improperly formatted like it used to be the case with nanode rf.

Maybe something that helps for next release...

I can't compare to the nanode.

The time is sent here:

https://github.com/Jerome-github/oem_gateway/blob/master/oemgatewayliste...

copied from the php script there:

https://github.com/emoncms/raspberrypi/blob/master/raspberrypi_run.php#L313

No one complained about the php script, as far as I know.

If you have a github account, you may want to file a bug for this with as many details as possible, otherwise I'll probably forget about it. If you don't, tell me, and I'll do it.