Upload old data to emoncms.org

I finally got access to my old energy usage data from my energy company and would like to upload it to emoncms.org for comparison. The data is just kWh/month so over the last year I only have 12 data points.

I've tried every single way to upload and update values to the feeds I've created using the commands below:

curl http://emoncms.org/feed/insert.json?id=90303&time=1440979200&value=533&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1438300800&value=549&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1435622400&value=676&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1433030400&value=686&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1430352000&value=921&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1427760000&value=786&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1425081600&value=836&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1422662400&value=934&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1419984000&value=882&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1417305600&value=1080&...
curl http://emoncms.org/feed/insert.json?id=90303&time=1414713600&value=977&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1412035200&value=793&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1409443200&value=705&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1406764800&value=544&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1404086400&value=503&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1401494400&value=667&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1398816000&value=832&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1396224000&value=828&a...
curl http://emoncms.org/feed/insert.json?id=90303&time=1393545600&value=694&a...

For some reason it does not work. Please advise how I can import data into emoncms.org

Echo's picture

Re: Upload old data to emoncms.org

Same here. Uploading to a local instance returns only a "0".

I try to upload value for a daily feed of type PHPTIMESERIES. So basically the same setup as above.

Thanks René

jesjimher's picture

Re: Upload old data to emoncms.org

If I remember well, with PHPFINA/PHPFIWA you can't insert values older than the first value recorded. So, if you create an input today with current W values, it's not possible to insert historical values from older times. Look at emoncms.log, it's probably outputting errors there.

jesjimher's picture

Re: Upload old data to emoncms.org

In fact, a possible workaround would be inserting the first value of the feed as an artificially old value before inserting real data, so then historical values can be inserted.

For example, let's say that we create a new input/feed today for present values, but we would like to insert historical data from the last 6 months there, too. The steps could be:

  1. Insert an arbitrary value (i.e. 0) for 1 year ago: curl http://emoncms.org/input/post.json?node=0&json={newinput:0]&time=1420070400&apikey=whatever. (1420070400 is 20150101)
  2. A new input, named "newinput" should show at input list. Configure it to log its values to the feed you would like.
  3. Proceed to insert old values with repeated calls to emoncms.org/input/post.json?time=xxx&...

Perhaps this way of creating feeds should be the default, because inserting historical data is a very typical need.

Echo's picture

Re: Upload old data to emoncms.org

My workaround was to use MySQl data storage instead of PHPFINA/PHPFIWA. At least for the data feeds this was an option. I still have no solution for the calculated feeds, that are generated from new input values like "kWh to kWh/d".

I agree. Inserting old data is a common usage scenario. It should be considered.

René

ChrisValentine's picture

Re: Upload old data to emoncms.org

How did you force it to use mySQL? The comments in settings.php are very misleading. It says to "place a comma as the first character on uncommented lines but the first" but if you do that you get errors like:

PHP Parse error:  syntax error, unexpected ',', expecting ')' in /www/settings.php on line 38

pb66's picture

Re: Upload old data to emoncms.org

If you want emoncms to use a particular engine you do not want to uncomment it. 

Uncommented engines are hidden, therefore not available, so the commented out engines are available.

Paul

 

ChrisValentine's picture

Re: Upload old data to emoncms.org

Inverse logic. Daft, but it works!

fredrike's picture

Re: Upload old data to emoncms.org

In fact, a possible workaround would be inserting the first value of the feed as an artificially old value before inserting real data, so then historical values can be inserted.

For example, let's say that we create a new input/feed today for present values, but we would like to insert historical data from the last 6 months there, too. The steps could be:

Insert an arbitrary value (i.e. 0) for 1 year ago: curl http://emoncms.org/input/post.json?node=0&json={newinput:0]&time=1420070400&apikey=whatever. (1420070400 is 20150101)
A new input, named "newinput" should show at input list. Configure it to log its values to the feed you would like.
Proceed to insert old values with repeated calls to emoncms.org/input/post.json?time=xxx&...

Thank you!

 

Created a gist with my python code used to parse a csv-file with timestamp;value.

 

https://gist.github.com/fredrike/e825f2c7e6158e44e508

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.