Hi all
I now have over 1000 Solar inverters sending me data. When I started I used the approach to create an input and name the values nicely so that a normal human beeing is able to read the data. I use EmonCMS for monitoring micro inverters of Enecsys.
The input will look like this:
Because of the success of using EmonCMS I now have thousands of feeds running and as said over 1000 inverters and hundreds of other data like house consumption etc. sending data to the system. I have good Hardware which at the moment does not suffer from performance issues but I want to be prepared for the future as there is maybe a bigger Solar plant with an additional 270 inverters waiting.
Is there a way to use the bulk update on the exisiting inputs? So that I can use one hit for multiple node id updates? Looking at the manual on how to use bulk it just says that you can use it in this format:
/input/bulk.json?data=[[0,16,1137],[2,17,1437,3164],[4,19,1412,3077]]
Using this of course creates just new keys instead of using the existing ones. A lot of my users are using the input page for checking non important data like Net Voltage or Net Frequency and do not want to log it. So for them, of course, a nice readable user friendly output would be desired.
If not possible where do I have to change it? Is it in the input_controller.php or is there more I need to change?
Thanks a lot in advance for all your help!
Andreas
Re: Bulk updates on existing inputs
Hello Andreas, what do your current API calls look like? It looks like you must be sending using the JSON-like format with input key:value {key1:123} you can post to multiple inputs by extending it like this:
{key1: 123, key2: 123, key3: 123, ...}
There are things that you can do for performance if you need it, such as putting the input requests straight into queues so the connection is released as fast as possible, then running input queue processors to do the processing. Its not something that's really documented, but the components are there:
replace the existing input controller with this:
https://github.com/emoncms/emoncms/blob/master/Modules/input/que_input_c...
then run the queue processor here:
https://github.com/emoncms/emoncms/blob/master/scripts/input_queue_proce...
You need to keep an eye on all of that though, as they can fail, and then you'd have to restart the processor etc. Using Redis and SSD drives makes a big difference too. I imagine your doing that.
Re: Bulk updates on existing inputs
Hi Trystan
Yes, you're right. I use the JSON like format:
/input/post.json?node=200017088&json={AC_Output:165,DC_Input:174,Net_Frequency:60,Net_Voltage:250,Inverter_temp:40,Lifetime_Output:505,Alarm_state:0,Status:0}&apikey=xxxxx
So my question really is on how to send multiple nodes in the same JSON. The Bulk does it but I'm not able to use the KEY variable and it therefore always creates new ones.
I receive one big JSON string from the gateway with all the inverter data from multiple inverters and it would be really nice to also send it like this to emoncms. I could imagine that this would also be something from big use for others not just me.
An Example of how many inputs it could be is a view like this:
They all then send data as posted above as keys. I now have over 1000 doing that. My 24 Core Windows Server handles that load of course without any sweat but I expect to come more every day. Normally 2-3 users will come daily and bring an additional 10-40 inverters to the system :-)
I love how EmonCMS works and handles the stuff but this mass update thing is the most missing part for me at the moment. Having 30-50hits a second is not uncommon but I do not prefer it. Our windows servers in the office were handling hundreds of hits a second but I like it efficient :-)