Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

I've been uploading data to emoncms.org for nearly a year. I didn't configure the local emoncms at the time.

I've now installed 8.5 using the SD card image on a Pi, and would like to log locally as well as upload to emoncms.org. I want to import all the inputs/feeds/dashboards from emoncms.org to the Pi.

I have installed the useful scripts folder and am using the backup.php script. It does appear to have downloaded the feed data to /home/pi/data/php* directories.

In the web browser it hasn't restored the inputs or other data for me. Is it supposed to?

When I run backup.php I get the following notices.

Notice: Undefined index: phpfiwa in /var/www/emoncms/Modules/usefulscripts/backup/backup.php on line 107

Should this script restore all the feeds/inputs and dashboards?

 

thanks!

Paul Reed's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

In short, no.

The script only downloads the feed data, and you would have to manually re-establish the relationship between inputs & feeds, as well as laying out your dashboards again.

If your local emoncms is not recognizing the downloaded data folders, you may wish to check the permissions assigned to them.

Paul

pb66's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

I understood it also backed up the inputs if backup.php is configured to "link_to_local_emoncms"

"Set $link_to_local_emoncms to true if you wish to access your data within a local installation of emoncms. Set $local_emoncms_location and $local_emoncms_userid to link to your local emoncms installation."

the inputs.php looks like it is written to insert the backed-up inputs into the local table and since the processlist is included, the links between inputs and feeds should be carried over.

The error message you are getting is probably due to using the "low-write" version included on an SDcard which will not have a phpfiwa directory, you can get around the "backing-up" error by adding the directory to recieve the backups but you will not be able to use them with the low-write version.

You can convert phpfiwa to phpfina quite easily using the phpfiwa_to_phpfina.php script, the script will also move them to the phpfina directory (but you may need to edit the feed "type" in the feeds table with phpmyadmin).

I haven't used the "linked" backup script myself but it appears to have all the bits to work except for dashboards and multigraphs, I would recommend creating a new user in the local emoncms with no inputs or feeds and use that account userid and local apikey to avoid any input issues caused by inputs with the same nodeid etc.

Paul

a234525's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

I started with a new user id on the local instance. I think it only lets me create one user.

Here are the settings in backup.php. There isn't anywhere for the local apikey that I can see. I have put in the write key for emoncms.org

Not sure what to try next?

 

    $remote_server = "http://emoncms.org";
    // NEEDS TO BE YOUR WRITE APIKEY (ELEVATED PERMISSIONS FOR FULL DATA EXPORT)
    $remote_apikey = "[mykeyhere]";

    $link_to_local_emoncms = true;
    $local_emoncms_location = "/var/www/emoncms/";
    $local_emoncms_userid = 1;
    $backup_inputs = true;

 

AllenConquest's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

To retrieve the inputs from emoncms.org you need to set the $backup_inputs to true.

I am getting another issue though. I have backed up all the data from emoncms.org and can see the feeds and inputs in my local emoncms. On the feeds screen I can click on the little eye icon to visualize the feed data. These all seems to be working except any feed that is a DAILY datatype (kWhd feeds). Any ideas why these are not working in 8.5 ?

Allen

pb66's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

v8.5 no longer supports SQL based feeds, Trystan wants to take emoncms in the direction of using queries to provide daily totals, see https://github.com/emoncms/emoncms#branches for info on variants.

I believe the v8.5.2 extended branch supports all feed types, but you need to add the "engines_hidden" section to the settings.php for v8.5.2 to function and comment out the engines you do NOT want to hide (so the array is empty to get all the feeds (upside down, I know)

Paul

AllenConquest's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

Hi Paul,

Is there an upgrade path for moving from a DAILY SQL based feed? I found the sql_to_phptimeseries_mv.php script and successfully used that to convert the feeds to phptimeseries. But I still can't get the visualizations to work. Am I missing something? I don't really want to loose all that historical data, or are you saying it can now be recreated using the normal readings (and if so how?).

Allen

AllenConquest's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

Hi Paul,

Some visualizations are working, such as the Stacked Solar with my kWhd feeds, but others such as bargraph are not. 

Allen

pb66's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

"Master (v8.4) - in the process of being replaced by the v8.5 branch. v8.5 will break older installations of emoncms that are using the older feed engines: timestore, phptimestore and mysqltimeseries, the upgrade procedure has not yet been written, if you have an existing system thats working then stay on this branch. If your setting up a new emoncms installation use the v8.5 branch."

This is from the "branches" link and it advises against upgrading to the v8.5 for existing users (local emoncms) I think using the data (backed up) from emoncms.org maybe puts you in the "existing" rather than "new" installation group.

v8.5 relies on producing the daily totals from a query at runtime, so no the feeds are not recreated as such, in that they are never written to disc, but they are recreated in part on the fly each time you view the graph.

It is discussed a fair bit in the forum but off hand I cannot point you to a specific place, sorry.

I know this will sound bizzare, but the "extended" branch is actually more stable and still uses the sql feeds, so in this instance the "extended" branch could be a stepping stone towards adopting the mainstream version and dropping the sql if you choose to, the support seems to be better for the low-write and extended versions too, the v8.5 seems to be getting left out a little.

Paul

AllenConquest's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

Hi Paul,

I have moved over to the Extended branch and downloaded the data again. Now the visualizations are working.

Thanks for your patience!

Allen

a234525's picture

Re: Installed 8.5, how do I backup/restore inputs and feeds from emoncms.org?

I still haven't been able to restore any inputs to my local instance. Anyone any ideas?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.