backfill data into emoncms phpfiwa feeds

I was wondering if it should be possible to backfill data into phpfiwa feeds.

I've read trough the code and noticed that by default the feed is allowed to accept data in the range of 5 years back and 48 hours in the future.

<quote>

$now = time();
$start = $now-(3600*24*365*5); // 5 years in past
$end = $now+(3600*48); // 48 hours in future
if ($timestamp<$start || $timestamp>$end) {
$this->log->warn("PHPFiwa:post timestamp out of range");
return false;
}

</quote>

I've done some experiments with my  data. I still had an old emoncms v5 (:blush:) with data in mysql feeds. So I quickly hacked a script together that reads the data from mysql and creates post.json url's to my new installation with a timestamp according to the timestamp in the mysql feeds table. Importing this data in clean/not used feeds works perfectly. However if I try to import them in an existing feed, the feed just turns to inactive untill the next realtime data is inserted and no data is recorded in the phpfiwa files (which are files with a 10min range btw). So I decided to create all the feeds from scratch and import all my data there (after exporting the feeds from my new installation to csv files, with another script I hacked together) After I imported the mysql data in the new feeds and allowed my sensors to start monitoring again, I noticed a gap of 3 days. I want to backfill this gap with data from the exported csv files, since they contain the correct data but for some reason the data is not being imported. The result of the posts is always OK.
A sample post :

http://<emocms-server>/emoncms3/input/post.json?apikey=<insertkey>&json={bureautemp:17.50}&time=1421992800
 

I know there is no node in the post, this is as expected. I have a script on one of my raspberry pi's that is reading a 1-wire bus spread throughout the house with temperature sensors in every room and I've written a script to inject the data directly in emoncms.

I have no idea why this data is not inserted. Php is not throwing any errors, so I'm baffled why it's not working. Any suggestions are welcome. The resulted image is attached.

pb66's picture

Re: backfill data into emoncms phpfiwa feeds

As I understand it, although the -5yrs+48hrs filter exists the -5yrs part won't actually have any effect on a feed until it has over 5 years worth of data. Since the "fixed timeseries" engines all count the number of datapoints (file size) since the feeds creation the earliest point that can be referenced is the creation date. 

When trying to insert data stamped prior to the creation date/time the "updated at" time in the SQL will  be updated to a historic time/date so that the update column will show inactive but the data cannot be saved by the engine as that time period cannot exist in the data file.

As for the 3 day gap, it may be an anomaly in the number of datapoints and file size. If for example you have 6 days worth of datapoints (6days x 24hrs x n points per hour) and you put that in a feed with a start date of 7 days ago, when you post new data to that feed the engine will subtract the files start timestamp from the current time to calculate the correct position for the new data, when it checks the feed and finds insufficient datapoints it assumes no data has been received in the last 24hrs and pads out the 6days to 7 with null datapoints before appending the new datapoint.

You didn't mention what version you are using now, if it's the buffered-write version you cannot insert data only append new data and if it's v8.4.0 there is a "quirk" effecting the use of node 0 or omitting the node id.

Not really any firm fixes in there, but hopefully some food for though perhaps.

Paul

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.