(ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Update: This image has been superseded by the emonSD-20Feb16 image. This thread is now closed. Please post on the new thread for issues regarding this latest image. 

------------------------------------------------------------------------------------------------------------------------------------------------

This release is 'apha' please read comments below before installing. If you require stable release see emonSD-17Jun15.

DOWNLOAD ALPHA: emonSD-12Feb16.zip (1.2GB)

USA Mirror: emonSD-12Feb16.zip

MD5 Checksum (zip): 67f1843588800eb65b8fbe2dd3412eb1

MD5 Checksum (.img): 7c4693eca23baab0beb8ba6e9ad99572

Here is the latest version of our emonPi / emonBase (RaspberryPi + RFM69Pi) pre-built SD card image. 

This post follows on from Steps towards an EmonCMS 9.0 SD Card Image + Upgrade path, the Low-write mode - your thoughts please thread and blog posts on node-RED & MQTT, openHAB and LightWaveRF OOK control.  

Key Features: 

Please report any bugs on this thread. 

Data Migration 

If you're moving from our previous pre-built-SD card, and have Emoncms data you want to migrate, this can be done using the backup module. Backup module is ready installed on this image. You will need to install it on an older image: 

See Emopncms backup module readme: https://github.com/emoncms/backup

Please report any issues to his thread. 

 

Storage & Read-only File system 

The image is based on Lightweight Raspbian Jessie which makes for a very space efficient stack. With everything installed (inc nodeRED & openHAB) total system size is 1.7GB compared to the previous ready-to-go emonPi image, which was over 2GB without nodeRED or openHAB.

As before, the filesystem is in read-only mode by default. discussion thread.

To put the filesystem in read/write mode, use the command: rpi-rw 
To return to read-only mode, use the command: rpi-ro

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.4G  1.7G  1.6G  52% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M  6.3M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
tmpfs            40M     0   40M   0% /var/lib/openhab
tmpfs            50M  204K   50M   1% /var/log
tmpfs           1.0M     0  1.0M   0% /var/lib/dhcp
tmpfs            30M   36K   30M   1% /tmp
/dev/mmcblk0p1   60M   20M   41M  34% /boot
/dev/mmcblk0p3  194M   32M  153M  17% /home/pi/data


The main data folder for emoncms is /home/pi/data this folder is always mounted Read/Write. A number of other folders are mounted in RAM (tmpfs), see /etc/fstab. A read-only file system presents challenges to some core Linux processes such as DHCP and NTP. A couple of DHCP & Resolv files are located in the writeable ~/data folder and symlinked to /etc and a custom /sbin/dhclient-script is used.NTP-backup (thanks Paul) is used to enable NTP clock updates in R/O mode. See the original NTP discussion thread.

emonPi

Now updates itself on first factory boot so we will always ship with latest software pulled from github. emonPi LCD now shows when it's updating and also LCD scrip has been updated to include authenticated MQTT support and to display temperature, VRMS and pulse count values as well as power 1 &2.

 

Emoncms V9 

Access via: http://emonpi

Authenticate: Create the admin user account 

Configure: emonhub.conf (located in ~/data) via web interface 

Update: Pull in the latest updates from Emoncms stable branch using the web-based update tool in Administration > update emonPi.  

 

SSH  

The default SSH password has been changed from "raspberry" to "emonpi2016" since even though we recommend users change their password, not many do. Moving away from the default raspberry will give a little more security. It's still HIGHLY RECOMMENDED to change the SSH password using passwd (be sure the filesystem is in R/W mode before you change the password (rpi-rw) 

 

MQTT

Mosquitto MQTT server is running on port 1883.

emonHub posts emonPi data to topics:

emonhub/rx/<nodeID>/values topic.

​and

emon/<nodename>/<keyname> e.g. emon/emontx/power1

See here for more info an emonHub MQTT topics

Port 1883 is open by default and authentication has been turned on:

mosquitto username "emonpi"

mosquitto password "emonpimqtt2016"

to change password : 

sudo nano /etc/mosquitto/mosquitto.conf

add lines 

allow_anonymous false
password_file /etc/mosquitto/passwd

To generate a password file, use the mosquitto_passwd tool. It will prompt for a password, then create a hashed password file. 

$ sudo mosquitto_passwd -c /etc/mosquitto/passwd username
 
Note if mosquitto authentication credentials are change the new credentials will also need to be updated in:
~/data/emonhub.conf
/var/www/emoncms/settings.php
~/emonpi/lcd/emonPiLCD.py
~/oem_openhab/configuration/openhab.cfg
in node-RED MQTT node via web flows editor 
 

On this image the Nodes module has been depreciated in favor of the mqtt_input script. When data from a particular node is received (as long as it has a relevant entry in emonhub.conf) it will be posted to Inputs when it can be then be logged go feeds. Any data posted to the emon/# MQTT topic will be logged to feeds using tbe topic names as Input name and Key name, e.g: emon/emontx/power1 will result in an input called emontx with a key called power 1 and the value. 

The mqtt input script can be stopped, started and restarted using:

sudo service mqtt_input restart 

The mqtt_input scripts logs to the emoncms log which can be viewed in Admin > logger. The default log level is WARN this can be changed to INTO for a more verbose log in emoncms settings.php. The MQTT server authentication details used by the script are also set in settings.php. 

 

NodeRED

Node-RED is installed and set up with a basic (but useful!) demo flow to subscribe to emonHub MQTT topic and send an email to alert the user when emonTH batteries are low. For email to work, you will have to enter your gmail or smtp details into the node-RED email node. See node-RED blog post. For  our install and setup notes see oem_node-red repo

Access Viahttp://emonpi:1880

Authentication: user "emonpi" and password "emonpi2016" 

The User folder is in the R/W partition so flows can be saved via web editor without he need to change to R/W mode /home/pi/data/node-red

Set user name and password in /home/pi/node-red/settings.js. See instructions on out nodeRED github

https://github.com/openenergymonitor/oem_node-red

 

Note: Node-RED as setup in the image is compiled for Pi2, to use with Model B+ Node will need to be recompiled. Follow these instructions: 

Check if node & npm are installed/working properly, normally you will get these results:

rpi-rw
​sudo apt-get remove nodered
pi@emonpi:~$ node -v
Segmentation fault
pi@emonpi:~$ npm -v
Segmentation fault

Install ARM v6 (for a RasPi model B 512MB) LTS version of node.js

wget https://nodejs.org/dist/v4.2.4/node-v4.2.4-linux-armv6l.tar.gz
tar -xvf node-v4.2.4-linux-armv6l.tar.gz
cd node-v4.2.4-linux-armv6l/
sudo cp -R * /usr/

Check if Node-Red is working (can take up to 40-50s on a RasPi model B 512MB)

sudo service nodered start

Re-enable the Node-Red service (maybe not nescessary, haven't tested it)

sudo systemctl daemon-reload
sudo systemctl enable nodered.service

Check if Node-Red gets auto-started on boot:

sudo reboot

 

OpenHAB 

OpenHAB is installed and setup with the basic skeleton OpenEnegyMonitor config. See openHAB blog post.

Access viahttp://emonpi:8080

Authenticate: user "pi" and password "emonpi2016"

Change user name and password in /etc/openhab/configurations/users.cfg

Configure: oem.items and oem.sites and located in git repo ~/oem-opemhab and symlinked to default.items and default.sitemap in relevant folders in to /etc/openhab/configurations. 

Note: Config files are symlinked to our oem_openhab git repo so we can add functionally with git updates, pulled in automatically with emonPi update. It's recommended to setup your own sitemap and files config so your changes will not be overwritten by out updates:

e.g. dave.items and dave.sitemaps then navigate to http://emonpi8081/openhab.app?sitemap=dave

 

LightWaveRF OOK RF Tx 

Service to control lightwaveRF plugs and relays via OOK (OOK Tx module installed on emonPi) controlled via MQTT and openHAB. See LightWaveRF OOK blog post.

Timezone 

The timezone can be set by running raspi-config then selecting 'Internationalisation Options':

sudo raspi-config

Caution: don't try and run the 'Expand file-system option' in raspi-config. This won't work because of the extra third partition.  

  •  
Jon's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Glyn - I was reading through all of the above and when I got to the Node-RED section it dawned on me I don’t know how to backup and restore my Node-RED flows and credentials.  Can this be added to the “Emoncms backup module” at: https://github.com/emoncms/backup sometime in the future?  

Or if this is simple can you point me in the right direction?  I’ve done a google search for "back-up restore node-red" and there are lots of different suggestions but I’m not sure which is correct.  (but I do plan on figuring it out!)

This maybe needed for others for OpenHAB.

Jon

PS - Just want to say "thank you" to all for the hard work and dedication.  You guys go above & beyond in your support and kindness to the newbies like me! 

Bramco's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

+1 for the thank you and above and beyond...

1.2G is great as well Glyn, at least for cheapskates like me that have plenty of old small SD cards around  :-)

Well done.

Simon

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Great work, can you advise how to disable both the Lightwave Module and also the LCD script as I need all the GPIO pins for Openhab use on my pi. On the previous image just disabling the LCD service didn't stop it taking one of the GPIO pins at boot..

thanks

mrpetrov's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Hi, Glyn, folks!

The instructions for the error that happens when upgrading from the previous December image proved usefull.

But... this being BETA - I'm running the "continuous sampling" code on my EmonPi, and just running Administration->Update, restart, ssh in emonpi and updating to CS atmega firmware doesn't work well: the text on the LCD screen contains strange symbols, and the backlight doesn't work as it is supposed to. Otherwise everything is fine.

With that out of the way - why not switch all things BETA to "continuous sampling" or it is not as simple as I think it is?

Thanks!

Plamen Petrov

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Thanks for your kind words of support. It's still a work on progress but I feel that we're nearly there with this. It's been a long journey. 

Exporting node red is not too difficult. In the editor just select all then choose export to clipboard and copy the resulting text. Then simply select import and paste into the new version. It's a good idea to save this text to a file as backup. 

 

'Sudo service lwrfd stop' will stop Lightwave RF mqtt service. 

 

We are still testing continuous sampling, we did initially has some memory instability issues. However the latest V3.2 update seems to have fixed this. Yes, reckon we will go for pointing towards continuous sampling in the beta version soon. What has been your experience with continuous sampling code? 

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

What issues are you experiencing with the LCD? I will test with continuous sampling tomorrow.

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

I still can't create new feeds with the new image. 

As the backup module wouldn't show on my old image after following the instructions, I had to copy the data files over manually by:

1)Stopping Feedwriter
2)Expanding the data partition on the card after it ran out of disk space by loading it up in GParted
3)Restarting the Pi and copying the feedfiles into a temporary folder with WinSCP
4)sudo copying the feedfiles from the temporary folder to /home/pi/data/phpfina
5)CHOWN the files to www-data
6)redis-cli "FLUSH ALL"
7)SQLDump the old database and copy to new machine with WinSCP
8)Import the database into the new images using the command line from the backup module
9)Update the database from admin in the web interface
10)Start feedwriter and after discovering the dashboard is frozen, reboot
11)When its restarted, observe usual dashboard updating and feeds page showing correct update times together with inputs page working

I then tried to create a new feed, first by:

1)Add Kwh/d onto new feed
2)Feed Undefined error appears in a popup, and after clicking OK, I discover the feed page is broken again.

So to try and get sensible logging out I:

1)Turned on Logging
2)Disabled the Mosquitto Bridge so there was no incoming data, to keep the log clear of noise
3)Stop Feedwriter service
4)reimported the SQL Database
5)redis-cli "FLUSHALL"
6)Start Feedwriter service
7)reboot
8)check its all working again with the original number of feeds
9)find an unused input, select log to feed with all the default settings and press OK.
10)"ERROR: Feed could not be created, undefined." appears in a popup
11)Click OK
12)Discover feeds page is broken again
13)Write this post and add the log to it

The only change I have done to the image apart from expanding the data partition is setting my location and timezone using raspi-config, is adding a bridge to mosquitto so I can get some live data into the image for testing.

I also checked that /var/lib/redis was empty and it was

 

Xino's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Also getting the "Feed Undefined error popup" when creating new feeds.

One change in common to the image was expanding the data partition. I used the sdpart script. 

 

Another issue ...

Followed above instructions to install nodered on a Model B+ RPi but then get:

pi@emonpi:~/data/tmp/node-v4.2.4-linux-armv6l $ sudo service nodered start
Failed to start nodered.service: Unit nodered.service failed to load: No such file or directory.

 

Paul Reed's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Luis - quick question (and slightly off topic!) - do you have any feedback on your experience of using the sdpart script, did you have any difficulties or encounter any issues?

Paul

TrystanLea's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Trying to debug this here, after a feed create error, is anyone else seeing segmentation errors in:

sudo tail -f /var/log/apache2/error.log

i.e:

[Sat Feb 13 18:17:40.611618 2016] [core:notice] [pid 14893] AH00052: child pid 15562 exit signal Segmentation fault (11)

 

Jon's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Looking for help with Backup (export) from a emonSD22Dec15 image so I can move to the emonSD12Feb16 image.

I’m in the Data Migration section above and I am following the steps for the Emoncms backup module readme at https://github.com/emoncms/backup.  The install seemed to be fine and the new Backup webpage appears.  But when I click the Create backup button there is only once response on the virtual Terminal window.  It states Backup flag set and then nothing.  No backup file is created in the /home/pi/data directory.

The config.cfg file looks correct and is the same as emonpi.default.config.cfg file. 

I can launch service-runner by itself with /home/pi/backup/service-runner and the backup seems to run OK.  

If I launch service-runner with output going to the service-runner.log then I get a permission error.

pi@emonpi:~$ /home/pi/backup/service-runner >> /var/log/service-runner.log 2>&1
-bash: /var/log/service-runner.log: Permission denied

 

So I am guessing service-runner is not running properly in crontab and I do not know where to look for these errors. Again I am just guessing, but should it be launched from the root crontab (sudo crontab?)?

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

We're working on the feed creating error...

@sheppy 

Regarding installing the backup module on your old Pi, I have just updated the readme with some more detailed instructions. In particular I've just corrected a mistake in the backup install instructions 

ln -s /home/pi/backup /var/www/emoncms/backup  has been corrected to ln -s /home/pi/backup/backup /var/www/emoncms/backup. Notice the extra backup folder depth. You can fix this on your old pi by deleating the symlink then creating another 

sudo rm /var/www/emoncms/backup

ln -s /home/pi/backup/backup /var/www/emoncms/backup

As mentioned on the install insturctions make sure have have the latest version of Emoncms on stable branch and restart apache to enable symlinks. See full backup module install instructions

Jon's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Glyn - How do I make sure I have the latest version of Emoncms on stable branch?  Is there an easy way to check what branch I git (Stable vs Master vs something else)?

Paul Reed's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

From your emoncms directory;

git branch -a

...will show you which branch you are on.

If it's 'stable';

git pull

Paul

Jon's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

I am assuming this is all OK...  (see attached image)

What is the reference at the bottom to "master"?

From https://github.com/emoncms/emoncms
   b306c61..3cfa332  master     -> origin/master

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Yep, got that error the moment I clicked the add button on the new feed and before clicking OK on the error message

[Sun Feb 14 10:28:51.812926 2016] [core:notice] [pid 1105] AH00052: child pid 1173 exit signal Segmentation fault (11)

The log has no further entries until I clicked on the Feeds tab and then I got

[Sun Feb 14 10:30:34.926240 2016] [core:notice] [pid 1105] AH00052: child pid 1174 exit signal Segmentation fault (11)
[Sun Feb 14 10:30:35.927686 2016] [core:notice] [pid 1105] AH00052: child pid 2351 exit signal Segmentation fault (11)
[Sun Feb 14 10:30:39.935921 2016] [core:notice] [pid 1105] AH00052: child pid 2348 exit signal Segmentation fault (11)
[Sun Feb 14 10:30:39.936255 2016] [core:notice] [pid 1105] AH00052: child pid 2350 exit signal Segmentation fault (11)
[Sun Feb 14 10:30:40.937523 2016] [core:notice] [pid 1105] AH00052: child pid 2352 exit signal Segmentation fault (11)

It then settles to adding the same error every 5 seconds or so whilst the browser is sitting on the blank feeds page. Clicking back on the inputs page makes the errors stop.

Comparing the 22Dec Image Apache2 Settings with the 12Feb settings yesterday I noticed a few differences but changing the 12Feb Image to be the same as the 22Dec didn't cure it, assuming I got them all. I found differences in files in:

/etc/apache2/ennvars
/etc/apache2/sites-enabled/working000-default.conf
/etc/apache2/apache2.conf

I've also tried copying the emoncms folder from the working one to the non working one and it didn't change anything. I did that from Samba so permissions could have changed when I did it. Obviously I restored the image before I did other tests.

Paul Reed's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Jon, that looks fine, you are now up to date on the stable branch.

Paul

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

In the build guide at https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/readme.md

it mentions

"For <Directory /> and <Directory /var/www/> change AllowOverride None to AllowOverride All. This should be on lines 155 and 166 of /etc/apache2/apache2.conf"

This hasn't been done on the 22 Dec Image, changing it back to defaults sadly, at least on my system doesn't seem to fix it.

One other thing I have noticed simply because my test system has a monitor on the end of it, is this which may be causing problems:

Feb 14 11:06:10 emonpi systemd[1]: Starting /etc/rc.local Compatibility...

Feb 14 11:06:10 emonpi redis-server[753]: *** FATAL CONFIG FILE ERROR ***
Feb 14 11:06:10 emonpi redis-server[753]: Reading the configuration file, at line 103
Feb 14 11:06:10 emonpi redis-server[753]: >>> 'logfile /var/log/redis/redis-server.log'
Feb 14 11:06:10 emonpi redis-server[753]: Can't open the log file: No such file or directory

Feb 14 11:06:37 emonpi systemd[1]: rc-local.service: control process exited, code=exited status=127
Feb 14 11:06:37 emonpi systemd[1]: Failed to start /etc/rc.local Compatibility.

Thing is there is an empty apache2 directory in /var/log which had to be being created by something, so maybe that error is spurious and the file gets loaded later.

 

 

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

You are on Stable branch. The little * indicates this. Stable is the default branch for emonPi images and recommended for normal users health and happiness :-). It usually only lags the master branch by a week or so. Just giving us chance to test new changes.  

Thanks for the logs, we're continuing to debug this. It's strange the issue only effects Inputs posted from HTTP rather than posted from the mqtt_input script. This is why we did not notice this sooner, I had mainly bee testing with inputs posted from emonHub via mqtt_input script 

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Could I ask you to check if the feed creation issue persists after a factory reset. Caution: you will loose the Emoncms account any any feed data

$ rpi-rw

$ cd ~/emonpi

$ sudo su

$ ./factoryreset

Let the Pi reboot 

I've just done a factory reset and I can now create feeds from HTTP inputs 

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@glyn,

I just did the following:

reset it,
reinstated my mosquitto bridge so I had inputs to create feeds from
created a new user, set the timezone & updated the database
created 12 feeds using default options, and all looked OK
then rebooted
feeds list is broken after the reboot

I did this twice so its very repeatable, something bad happens during the reboot

I also had to restart the mqtt_input service after each reboot before the inputs would update

In case you want to recreate the bridge too I've included the details below, its useful to test from a live system without adding a RFM69Pi

This goes at the bottom of /etc/mosquitto/mosquitto.conf and my live system is on a private network so it has an open mqtt server

connection oldimage
try_private false
address 192.168.1.10
start_type automatic
local_username USERNAME
local_password MQTTPASSWORD
clientid newimage
notifications true
topic # in 0 emon/ emon/

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@Glyn Just reading through this thread again, aren't my inputs created by mqtt_input? They stop updating until I restart mqtt_input after a reboot.

ALSO

@Glyn after reading @ Jon's post about service-runner permission errors and reinstalling the backup and link, I now have a working backup after I changed the crontab -e entry to read

* * * * * ~/backup/service-runner >> /home/pi/data/service-runner.log 2>&1

 

 

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@Jon Sorry I missed you post earlier regarding service-runner.log location. Thanks @sheppy for bringing to attention. I have added option in backup readme to point the log to ~/data which does not require any permissions. If using /var/log (which is my preference) the log file and permissions need to be created on startup using /etc/rc.local since the entire /var/log is mounted as temp fs in RAM on the emonPi. See example in emonpi rc.local:

https://github.com/openenergymonitor/emonpi/blob/master/rc.local_jessiem...

@sheppy I have come to the same conclusion that it's the reboot not the source of the input (HTTP/MQTT) which determined if the feeds are saved or not. We're still investigating this. There is something strange going on here with Apache... 

jason's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Hi Glyn

Just trying this beta for the first time - and I'm new to the forum. 

I started with the clean image, and immediately tried to load in my data from a backup - the result was not being able to log into emoncms at all. I noticed that "data/emoncms.conf" was empty so I did a factory reset ...

After the reset / reboot, I re-registered a user and logged into emoncms. I still cannot create a feed.

I can do some more debug if you like.

Bugs found during this process below :-) 

Jason 

BUG: When selecting the backup filename, the code will allow a filename of "backup.tgz" - which then fails to import. Renaming to "backup.tar.gz" will work !

BUG: emonpi password should be reset on "factory reset"

BUG: Cannot add feeds following factory reset

 

Paul Reed's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Could someone experiencing this problem please tell me what you get if you;

ls -la /etc/rc.local

and also

ls /home/pi/emonpi/rc*

I'm asking because I think theres a typo in the SD Card Build guide in section 8.

 

This appears to be incorrectly creating the symlink, ie emonpi/emonpi instead of emonpi, which would mean that rc.local (which creates the folders and log files in tmp following a reboot) is not loading.

Thanks

Paul

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@glyn backup changes completed and it now works. My old image was using the rc.local linked from /var/www/emoncms/scripts as described in the build guide - now all corrected to use the updated one from your link. I also manually reconfigured logrotate to use the version in emonpi, as last time I did a git pull on emoncms it moaned about logrotate, and after I stashed it & updated it broke my live system until I added it back.

@paulreed which issue are you chasing? I have the feed creation problem, and ls -la /etc/rc.local gives:

lrwxrwxrwx 1 root root 38 Jan 27 23:50 /etc/rc.local -> /home/pi/emonpi/rc.local_jessieminimal

rc.local no longer errors on the monitor during bootup since @glyn's update, but it did with the 12/2 image before

ls /home/pi/emonpi/rc* gives:
/home/pi/emonpi/rc.local  /home/pi/emonpi/rc.local_jessieminimal

​Both commands are on a factory reset version of 12/2 so it would have had the update on first boot run, and I've also updated /home/pi/emonpi/rc.local_jessieminimal by copy and pasting into nano, but haven't touched the symbolic link

 

Jon's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Paul - I get:

pi@emonpi:~/data/myApps$ ls -la /etc/rc.local

lrwxrwxrwx 1 root root 24 Dec 15 09:50 /etc/rc.local -> /home/pi/emonpi/rc.local

pi@emonpi:~/data/myApps$ ls /home/pi/emonpi/rc*

/home/pi/emonpi/rc.local  /home/pi/emonpi/rc.local_jessieminimal

pi@emonpi:~/data/myApps$ 

 

Paul Reed's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Thanks.

I stumbled upon the typo whilst looking for something else, obviously that has now been sorted - but not in the build guide.

I'll send a git pull request to correct it.

Paul

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@glyn really pleased you can recreate the problem, let me know if you need anything testing. I'm 13 hours ahead of the UK right now so please consider me the night shift. Currently doing paperwork so any excuse to be distracted gratefully received!

stuart's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Just to let you know that I installed this version on a PI2 yesterday without any problems.

The build has come on a long way in the past few versions - well done!

I'm planning to get back into the "monitoring thing" as I've got a bit more spare time now.

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

We're struggling to get to the bottom of why apache is throwing segmentation errors ans having trouble loading creating and loading Feeds. These are the sorts of errors I can see in /var/log/apache2/error.log (or /var/log/error.log if you have not yet updated to latest version). 

[Mon Feb 15 23:27:21.274566 2016] [core:notice] [pid 1261] AH00052: child pid 2097 exit signal Segmentation fault (11)
[Mon Feb 15 23:27:21.274728 2016] [core:notice] [pid 1261] AH00052: child pid 24964 exit signal Segmentation fault (11)
[Mon Feb 15 23:27:21.274893 2016] [core:notice] [pid 1261] AH00052: child pid 24966 exit signal Segmentation fault (11)

Can you confirm that these errors are the same that you are experiencing? It's strange that the Dec22 image that I have been running totally stable since December does not have this issue and is running the same version of apache, MYSQL and redis and should have the same config. The only difference I can see is   PHP version :

Dec22 image: 5.6.14-0+deb8u1 (Zend Version 2.6.0)

Feb16 Image: 5.6.17-0+deb8u1 (Zend Version 2.6.0)

I have changed to name of this thread from beta > alpha to alert folks to the issues this build is experiencing.   

 hanks you so much for helping with debugging, I know how much time this can absorb! 

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@jason 

Thanks for your feedback: 

BUG: When selecting the backup filename, the code will allow a filename of "backup.tgz" - which then fails to import. Renaming to "backup.tar.gz" will work !

Fixed.

BUG: emonpi password should be reset on "factory reset":

Fixed 

BUG: Cannot add feeds following factory reset

Still working on this one, see post above... 

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@glyn I did a search on signal segmentation fault and it may be something to do with PHP memory errors. - Maybe something is going into an infinite loop until it runs out of memory and spits the error? Just a thought.

See 
http://unix.stackexchange.com/questions/199561/apache-and-php-not-workin...
from just under a year ago

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@Glyn, I get the same errors - see my post above dated 14/02/2016 - 09:39.

A quick google suggests that error message can be logged by PHP out of memory errors so maybe it goes into a continuous loop until it runs out of memory and spits the error, meaning the error is a symptom, rather than the cause of the problem?

As the problem appears when creating new feeds on a brand new installation after a reboot, could it be something isn't being saved properly during a reboot?

(Apologies for the duplicate content, the first post vanished without a trace and left no clues about where it'd gone behind)

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

We have a solution! 

It turns out that php5-redis needed to be updated to work with the latest version of php. The latest version of php5-redis available through Debian Raspbian Jessie and currently installed on the emonSD-12Feb16 image is 2.2.5-1.

You can check php5-redis version in the sources with 

sudo apt-cache show php5-redis

Installing the latest stable version of redis (2.2.7) direct from phpredis github fixed the issue. Here are the steps to update php5-redis: 

git clone --branch 2.2.7 https://github.com/phpredis/phpredis
cd phpredis
​cat php_redis.h | grep VERSION
phpize
./configure 
sudo make 
sudo make install

Please could you confirm this fixes the issue for you? I will release another image with this fix. 

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Fantastic news! I'll test it later today whilst the UK is in bed and report back.

Jon's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

I don't have the apache segmentation errors above - I haven't gotten that far in the upgrade.

I cannot get Import Backup to work on the emoncms Backup webpage for the emonSD-12Feb16 alpha image. I get the error “There was an error uploading the file, please try again!”.  So I added the emoncms-backup-2016-02-16.tar.gz file to the /home/pi/data/uploads directory and ran emoncms-import.sh.  If this is an appropriate work around, then we may need more space for /home/pi/data.  

I ran emoncms-import.sh and ended up with lots of “No space left on device” errors.

=== Emoncms import start ===
Tue Feb 16 11:38:15 CST 2016
Reading ~/backup/config.cfg....
Location of mysql database: /home/pi/data
Location of emonhub.conf: /home/pi/data
Location of emoncms.conf: /home/pi/data
Location of Emoncms: /var/www/emoncms
Backup destination: /home/pi/data
Backup source path: /home/pi/data/uploads
Starting import from /home/pi/data/uploads to /home/pi/data...
Image version: emonSD-12Feb16
new image
Backup found: emoncms-backup-2016-02-16.tar.gz starting import..
Decompressing backup..
Restore phpfina and phptimeseries data folders...
cp: error writing ‘/home/pi/data/phpfina/1.dat’: No space left on device
cp: failed to extend ‘/home/pi/data/phpfina/1.dat’: No space left on device
cp: error writing ‘/home/pi/data/phpfina/18.dat’: No space left on device  ...

Here is the disk space available after the above error:

pi@emonpi:~/backup $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.4G  1.7G  1.6G  52% /
. . .
/dev/mmcblk0p1   60M   20M   41M  34% /boot
/dev/mmcblk0p3  194M  194M     0 100% /home/pi/data

For what it is worth I noticed on the emonSD22Dec15 image the /dev/mmcblk0p3 or /home/pi/data size was 477 MB (vs. 194 MB and on the emonSD-12Feb16 image )

pi@emonpi:~$ df -h | grep "^/dev"
/dev/root       3.1G  1.5G  1.4G  52% /
/dev/mmcblk0p1   61M   20M   42M  32% /boot
/dev/mmcblk0p3  477M  205M  248M  46% /home/pi/data

 Let me know if more detail is needed.

Xino's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@glyn...Installing the latest version of redis (2.2.7) fixed the feed segmentation bug. Great support. Thank you! 

BR

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@Jon how big is the backup file you are trying to restore?! Looking at my Emoncms account I have a feed with 6 years of power data @ 10s intervals is 65Mb. 

The ~/data partition was sized so that it would fit on a 4GB SD card image. If you have access to a larger card you could use Gparted on Ubuntu Linux or equivalent to expand the partition size. 

I would recommend you implement the php5-redis error fix anyway. I am currently uploading an emonSD16Feb16 image with the latest updates and the php5-redis bug fix. I will post the new image on another 'beta'  thread tomorrow morning.

@Xino your welcome :-) .Happy to hear it's working for you. It feeliks like it's been a long journey to get this this point, I feel like we're nearly ready for a 'release' image :-D. Thanks again for all your helping to debug. Thanks must go to @trystanlea for helping me to identify and fix the segmentation errors php5-redis bug.   

Jon's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@Glyn - the emoncms-backup-2016-02-16.tar.gz is 33 MB and the uncompressed is 99 MB.  My emonPi emoncms is less than 1 year worth of data @ 10 sec intervals except for environmental items that are 1 or 2 min intervals. I will confirm the intervals and scan for feeds I can delete.  Tnx!

 

sheppy's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@Glyn, I can now create feeds on both of my test builds and also on my live system - thanks very much!

I too have the "There was an error uploading the file, please try again!" problem when trying to restore a backup to the latest image. I have 3 gig free diskspace on the 4GB data partition, with a 47MB Backup file.

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

New image has been posted (emonSD-16Feb16) including the latest changes: 

http://openenergymonitor.org/emon/node/12189

Would you mind sending me via dropbox your backup file so I can try and debug the restore issue. A workaround would be to manually copy the file into ~/data/uploads

jason's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@glyn - no problem.

I'm now going to try a migration - assuming that other with more experience will be checking functionality. So, here's what happens:

1 - Start with Jun15 image, added backup, then manually ran emoncms-export.sh (did not spend time getting backup to work in the GUI)

2 - On a new SD, run the 16feb16sd image. Check that input is working, etc.

3 - Restore from backup in #1

4 - Confirm that inputs look ok :-)

5 - Confirm that feeds look ok ... er ... :-(

The Feeds page says "loading" and sticks there.

Try to create a feed from the input - message "ERROR: Feed could not be created, undefined"

Try to read a feed using the API "http://192.168.16.31/emoncms/feed/list.json" - and get a PHP error:

Fatal error: Uncaught exception 'Exception' with message 'ABORTED: Engine id '1' is not supported.' in /var/www/emoncms/Modules/feed/feed_model.php:71 
Stack trace: 
#0 /var/www/emoncms/Modules/feed/feed_model.php(396): Feed->EngineClass('1') 
#1 /var/www/emoncms/Modules/feed/feed_model.php(279): Feed->get_timevalue('10') 
#2 /var/www/emoncms/Modules/feed/feed_model.php(253): Feed->redis_get_user_feeds(1) 
#3 /var/www/emoncms/Modules/feed/feed_controller.php(38): Feed->get_user_feeds('1') 
#4 /var/www/emoncms/core.php(64): feed_controller() 
#5 /var/www/emoncms/index.php(146): controller('feed') 
#6 {main} thrown in /var/www/emoncms/Modules/feed/feed_model.php on line 71

 

... tried a reboot, same situation.

Also looked at logs in "Admin->Logger" and saw:

2016-02-17 11:58:42.903|WARN|phpmqtt_input.php|Subscribing to: emon/#
2016-02-17 11:58:42.904|WARN|phpmqtt_input.php|Not connected, retrying connection
2016-02-17 11:58:42.905|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0 
2016-02-17 11:58:42.906|WARN|phpmqtt_input.php|Subscribed to topic: emon/#
2016-02-17 11:59:18.463|ERROR|feed_model.php|EngineClass() Engine id '1' is not supported.
2016-02-17 11:59:22.067|ERROR|feed_model.php|EngineClass() Engine id '1' is not supported.

I will try to look at this - but any pointers would be useful.

 

glyn.hudson's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@jason 

Have you applied the update to php5-redis as described on my post above yesterday?

Or even better try the latest image (posted this morning) with the fix all ready included. Please post on the new thread for issues regarding the new image. 

It sounds like you are doing everything correct. I would be interested to hear how you get on with the new iamge. What size is the .tar.gz backup file you are restoring? 

 

jason's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

Now trying "factory reset"

So, from message above, did a factory reset, which should put everything back in place. So tests ...

1 - run factory reset under a root shell - ok

2 - login - password now the "factory" one :-)

3 - Check inputs - working :-)

4 - Go to feeds page - all seems ok :-)

5 - Back to inputs, hit the spanner to create a feed 

6 - Hit "add" and ... :-(

When I hit "Add", I get, in a dialogue box:

ERROR: Feed could not be created, undefined

What to do now ?

jason's picture

Re: (ALPHA): LATEST emonSD12Feb16 - emonPi / emonBase (Raspberry Pi + RFM69Pi) Emoncms v9 pre-built SD card image

@Glyn - this is all with the new image !

I'll check the php5-redis version ...

*** Moving to new thread ...

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.