Logfile - change of location

A number of recent changes has taken place to have greater control over emoncms logfiles, including being able to specify different levels of logging from within the emoncms settings.php, and now for emonbase users, the logfile location has been moved from the emoncms installation directory to /var/log/
The reasons for this are;

  • /var/log is the established location for logfiles, and where people expect them to be
  • Logrotate will then be able to manage the emoncms log, rotating, and limiting it's size
  • In low-write mode, /var/log is placed into tmpfs (RAM) and therefore emoncms logging can continue to run without disk-writes

Following a git pull,the implications for emonbase users are; (does not apply to emonPi - as emonPi already uses /var/log for it's logfile)

Existing Installations (not in low-write mode)
Emoncms is unable to create a logfile, only write to one, therefore it's necessary to create the logfile and set it's permissions in it's new location - just like what was done during the initial installation:

touch /var/log/emoncms.log
​chmod 666 /var/log/emoncms.log

Emoncms will then commence logging at /var/log/emoncms.log and be managed by Logrotate.

Existing Installations in low-write mode (Raspberry Pi's)
Following a git update, it will be necessary to run an update script, to make a number of required changes:

cd /var/www/emoncms/scripts/logger/ && sudo chmod +x install.sh
sudo ./install.sh

A more detailed explanation of what changes the script makes can be found here.

New Installations
The Raspberry Pi low-write guide has been updated to reflect these changes.

Feedback as always appreciated.

Paul

joyrider3774's picture

Re: Logfile - change of location

since i'm using windows i have no /var/log so i'm just mentioning the previous value that was in settings.php to make the log save inside emoncms installation folder in case anyone not running linux needs it. It was $log_filename = dirname(__FILE__).'/' . 'emoncms.log';

i have not tried it yet, but i do wonder where the log file would be saved if i just wrote $log_filename = 'emoncms.log'; will check it out once i update.

also i'm guessing logrotate is to make sure the log file size doesn't explode and i'm wondering if logrotate is not used (since i'm on windows) would the log file keep growing ? currently mine is 85 MB but i can't remember if there is / was something inside the php files code that would delete the log file if it had grown over a certain size.

Paul Reed's picture

Re: Logfile - change of location

Good point about the /var/log location in Windows, that needs to be covered within the Windows installation guide.

Regarding the logfile size - yes there is a 250mb limit set in Lib/Emonlogger.php but that is far too high when the logs are placed in tmpfs (low write), as only 30mb is allocated for the entire /var/log tmpfs directory to preserve RAM.

Logrotate really only applies to Raspberry Pi  installations, its not enforced in code, and unless you are directing your logfiles to tmpfs, it will use whatever Log Management System is installed by your OS by default.

In Windows, I doubt that you will notice any changes, other than the change of logfile location.

Paul

Jon's picture

Re: Logfile - change of location

Paul - I am not sure if this is related, but I get this error when running an emonPi Update.

Emoncms update started

Sat  2 Jan 18:43:16 CST 2016

git pull /var/www/emoncms
error: insufficient permission for adding an object to repository database .git/objects
fatal: failed to write object
fatal: unpack-objects failed

git pull /var/www/emoncms/Modules/nodes
Already up-to-date.

I tried rebooting the emonPi but got the same error.

Emoncms    Version    low-write 9.2 | 2015.12.11

emonPi formware 2.0

Paul Reed's picture

Re: Logfile - change of location

Jon, emonpi's should obtain their updates via the 'stable' branch, which has not been merged with development (master) yet. Also, these changes will not affect the emonpi's as they already write their logs to /var/log so these changes are really aligning the 'emonbase' logging to how 'emonpi's' already operate. So no, yours will be a separate issue, probably needing it's own thread. Paul

Jon's picture

Re: Logfile - change of location

Thank you Paul!

moved to new thread - emonPi Update error for emoncms (git pull /var/www/emoncms) http://openenergymonitor.org/emon/node/11926

Jon

glyn.hudson's picture

Re: Logfile - change of location

Hi Paul, 

Thanks a lot for packaging this up in a tidy way. I'm planning one more iteration on the new image before launch. I'll merge in these changes so the emonPi image setup will be inline with your RasPi Emoncms install notes.    

glyn.hudson's picture

Re: Logfile - change of location

Just tried with the new logfile, after running ./install.sh something seems to have gone wrong, my log file consists of:

[email protected]:/var/log$ cat  emoncms.log
2016-01-04 18:15:30.649|INFO|feedwriter.php|Starting feedwriter script
2016-01-04 18:15:30.791|INFO|feed_model.php|EngineClass() Autoloaded new instance of 'RedisBuffer'.
2016-01-04 18:15:41.729|INFO|feed_model.php|EngineClass() Autoloaded new instance of 'PHPFina'.
2016-01-04 18:15:41.760|INFO|PHPFina.php|lastvalue() 1
2016-01-04 18:15:41.777|INFO|PHPFina.php|lastvalue() 2
2016-01-04 18:15:41.788|INFO|PHPFina.php|lastvalue() 3
2016-01-04 18:15:41.798|INFO|PHPFina.php|lastvalue() 4
2016-01-04 18:15:41.807|INFO|PHPFina.php|lastvalue() 5
2016-01-04 18:15:41.816|INFO|PHPFina.php|lastvalue() 6
2016-01-04 18:15:41.824|INFO|PHPFina.php|lastvalue() 7
2016-01-04 18:15:41.834|INFO|PHPFina.php|lastvalue() 8
2016-01-04 18:15:41.843|INFO|PHPFina.php|lastvalue() 9
2016-01-04 18:15:41.852|INFO|PHPFina.php|lastvalue() 10
2016-01-04 18:15:41.860|INFO|PHPFina.php|lastvalue() 11
2016-01-04 18:15:41.877|INFO|PHPFina.php|lastvalue() 12
2016-01-04 18:15:41.886|INFO|PHPFina.php|lastvalue() 23
2016-01-04 18:15:41.892|INFO|PHPFina.php|lastvalue() 24
2016-01-04 18:15:41.899|INFO|PHPFina.php|lastvalue() 25
2016-01-04 18:15:41.907|INFO|PHPFina.php|lastvalue() 26
2016-01-04 18:15:41.917|INFO|PHPFina.php|lastvalue() 27

Have I missed something? 

Paul Reed's picture

Re: Logfile - change of location

That's strange, it must be something unique to the emonpi configuration...

Following a reboot on a new emonbase installation running Jessie, I get;

[email protected]:~ $  cat /var/log/emoncms.log
2016-01-04 18:34:11.646|INFO|feed_model.php|insert_data() feedid=2 updatetime=1451932436 feedtime=1451932436 value=411 arg=
2016-01-04 18:34:12.183|INFO|feed_model.php|EngineClass() Autoloaded new instance of 'RedisBuffer'.
2016-01-04 18:34:12.206|INFO|feed_model.php|insert_data() feedid=3 updatetime=1451932436 feedtime=1451932436 value=0 arg=
2016-01-04 18:34:12.224|INFO|feed_model.php|insert_data() feedid=4 updatetime=1451932436 feedtime=1451932436 value=0 arg=
2016-01-04 18:34:12.263|INFO|feed_model.php|insert_data() feedid=5 updatetime=1451932436 feedtime=1451932436 value=24179 arg=
2016-01-04 18:34:12.283|INFO|feed_model.php|insert_data() feedid=6 updatetime=1451932436 feedtime=1451932436 value=1250 arg=
2016-01-04 18:34:14.995|INFO|feed_model.php|insert_data() feedid=9 updatetime=1451932454 feedtime=1451932454 value=184 arg=

...and again following a reboot of another pi emonbase running Wheezy:

[email protected] ~ $ cat /var/log/emoncms.log
2016-01-04 18:41:40.161|INFO|feed_model.php|insert_data() feedid=13 updatetime=1451932899 feedtime=1451932899 value=332 arg=
2016-01-04 18:41:38.986|INFO|phpmqtt_input.php|Starting MQTT Input script
2016-01-04 18:41:40.426|INFO|feed_model.php|EngineClass() Autoloaded new instance of 'PHPFiwa'.
2016-01-04 18:41:40.427|INFO|PHPFiwa.php|PHPFiwa:post id=13 timestamp=1451932899 value=332
2016-01-04 18:41:40.633|INFO|feed_model.php|insert_data() feedid=14 updatetime=1451932899 feedtime=1451932899 value=332 arg=
2016-01-04 18:41:40.638|INFO|PHPFiwa.php|PHPFiwa:post id=14 timestamp=1451932899 value=332
2016-01-04 18:41:40.658|INFO|feed_model.php|insert_data() feedid=33 updatetime=1451932900 feedtime=1451932900 value=0 arg=
2016-01-04 18:41:40.681|INFO|feed_model.php|EngineClass() Autoloaded new instance of 'PHPFiwa'.
2016-01-04 18:41:40.682|INFO|PHPFiwa.php|PHPFiwa:post id=33 timestamp=1451932900 value=0

 

 

tgmaxx's picture

Re: Logfile - change of location

I am running a Wheezy configuration and noticed INFO msgs like the following are now appearing in my emoncms.log. Why am I getting these? Is there a possible problem?

2016-01-09 14:18:12.624|INFO|process_processlist.php|power_to_kwhd() feedid=61 last_kwh=0.0016374155555564 kwh_inc=4.5E-7 new_kwh=0.0016378655555564 last_slot=1452315600 current_slot=1452315600
2016-01-09 14:18:12.688|INFO|feed_model.php|insert_data() feedid=67 updatetime=1452349092 feedtime=1452349092 value=1 arg=
2016-01-09 14:19:12.689|INFO|feed_model.php|insert_data() feedid=60 updatetime=1452349152 feedtime=1452349152 value=0.018 arg=

 

 

Paul Reed's picture

Re: Logfile - change of location

When did you last update? 

I merged a fix a few days ago which sorts the issue.

If you update, you can then set your preferred logging level in settings.php (default is WARN).

Paul

tgmaxx's picture

Re: Logfile - change of location

Hi Paul, I just updated about 12 hrs ago so I have your latest commit. But I still don't understand the meaning of the msgs.

Paul Reed's picture

Re: Logfile - change of location

The messages are just what emoncms is doing under the hood. It's showing you what emoncms is doing with the data inputs, and calculating your daily values.
Im not at home now so can't check, but if you git pull, does it say that you are up to date - nothing to do?

tgmaxx's picture

Re: Logfile - change of location

Paul, I rebooted and I am not getting the messages anymore (with $log_enabled = true, and $log_level = 2). On git pull, I get "Already up-to-date."  Everything appears to be OK for now.

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.