Using curl to send a lot of data to the API

 Hi All,

I think I've nearly got this to work, but brain has frozen now, any help?

I have this big collection of historical data for many meters, going back several years, I want to upload.

I've modified the api_controller to allow me to post the time as strings.  It works.

Now I've got a big txt file with all my URL's ready to send to the API with curl:

The text file is formated like:

url="http://localhost/emoncms3/api/post?apikey=2da1475188840e0d9b8e7110297fb078&time=2012-08-15 23:45&json={metername:96.45}"

I'm trying to use curl -K mytextfile.txt

curl runs through the file, and the access log says they are 200 OK, but no data in the database.  But if I copy and paste one of the strings into the browser, it works.

Something about curl, I'm missing.

emdan's picture

Re: Using curl to send a lot of data to the API

 I found the problem was the space in my URL, between the date and the time, I replaced that with a %20 symbol and it worked.  Curl was only posting the URL up to the whitespace.

I'm trying to understand some logic behind emonCMS:

With emonCMS, you can't create the feeds, until the input starts sending.  There is no create input button.  In my case I need to send some dummy data to create the input, then setup the feeds, then send the data.  

For many people there is always going to be data lost, those packets that set up the input are dumped, until you set up a feed to log them.

You can't setup a daily data type feed, after the days have gone by, you need to set it up before hand.  Is that right?

emdan's picture

Re: Using curl to send a lot of data to the API

 *groan* ... now I can load all the data, feed_1 is real-time data and all its dates and values are correct.  The historial time values are set.

But feed_2 and feed_3, which are the Power to kWh/d, and histogram process, are logging everything with today's date, not the dates of the values its processing.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.