Archived Forum

The SQL database isn't the only storage mechanism (indeed relational databases for this sort of data are usually a poor choice) - look at the other storage options of OEM, such as phpfina

Ok, i did wonder because as you said relational DBs are bad for allot of sensor data.
I did just a quick read here: https://github.com/emoncms/emoncms/blob/master/docs/LinuxInstall.md
On the dependencies (but did not scroll)  and saw mysql. Now i also see Redis...
So where is the data stored?

Is there another way to feed the data into the DB beside making a HTTP request everytime something comes in?

What do you have against HTTP requests? Even with a number of data contributors you're not going to swamp a moderm webserver.

Why use a webserver if you could write directly to the database?
Modern hardware is not an excuse for bad practice ;-p
I have the opinion that everything needs to be done efficient as possible.

Don't get me wrong. Emoncms looks great and i know it is a lot of work to build something like that.
I just want to understand why things are done this way.

This is a big diversion:
MQTTbroker -> MQTTwarn -> Webserver -> DB

EDIT:
Ok just saw "MQTT changes 6th Feb": https://openenergymonitor.org/emon/node/12104
When i did google MQTT and Emoncms there was not much beside MQTTwarn.

But looks like just now somebody has the "Gotchas and Limitations" problem:
https://openenergymonitor.org/emon/node/12248

There is no problem connecting to a database and feeding it from another location/app...
But HiveMQ is Java and i know nothing about Java. If i would i could write a plugin.

Yes, good point, you could indeed send directly to mySQL from the emon Pi.

MQTT can be used with Emoncms: 

https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md

We use it by default on the emonPi: 

https://openenergymonitor.org/emon/node/12212

Thanks Glyn, but that was not the question.

Most proprietary database protocols - including MySQL - are heavier and more complex and tend to require specific layered APIs. HTTP and all the associated technologies are (usually) much lighter weight in terms of both sessions set-up and tear-down and implementation.

HTTP et al. are also far far far easier to debug on the wire.

I wrote a fair bit of documentation on the evolution of storing timeseries data in emoncms here which might be of interest https://github.com/openenergymonitor/documentation/tree/master/BuildingB...

The standard way of using MQTT with emoncms that we use is Mosquitto as the broker and then currently a php script with the Mosquitto-MQTT client (https://github.com/mgdm/Mosquitto-PHP)
that subscribes to the emon/emontx/power1 type topics. I think thats almost as direct as you can make it apart from cutting out emoncms features like the input and input processing layer.

Where did you read about MQTTwarn?

I can't believe it. After three days almost no sleep I did write my first Topic+Payload to MySQL directly from the Broker :-)

The plugin guide is well documented: http://www.hivemq.com/docs/plugins/latest/
I did take a look at the database example plugin (just found it even though I've used HiveMQ for 3-4 years) and saw that there is not much code in there: https://github.com/hivemq/hivemq-database-example-plugin
I never heard of Apache Maven before or did build jar files. So I gave it a try and did install Maven and IntelliJ IDEA.
I bet it would take an hour for a Java person to do the whole plugin to Redis.
But hey my head is already full - so it doesn't matter if there now needs to be also a minimum Java to get this up and running *lol

@TrystanLea, thanks i will check out your link.
3-4 years back i started with Mosquitto and also tried phpMQTT but then i found HiveMQ and it worked out of the box - especially WebSockets. When WebSockets came out it was a pain to get running before i used HiveMQ.
I was able to show a button push on an ATMega or switch a light from the browser right away.
I never logged data to a DB - just control things like my TV, Sat-Receiver with JavaScript over IR or control ZoomPlayer from any other PC, phone, tablet with MQTT.

I just want a nice graphical view of the data from a DB but building everything from scratch takes days, months, years. I also came across Node-Red and openHAB (that GUI is ugly *lol) and don't know if i like to have something running that is a single point of failure. I believe things should talk to each other without having apps running between them telling them: "if this then do that".
Ok there is the broker in between with MQTT - but thats the only thing and can be backed up with a little cluster.
Sure for people who buy commercial stuff they need something that translates all the protocols from the companies.

MQTTwarn example: https://openenergymonitor.org/emon/node/3634
Yes everything Google gave me was outdated and involved MQTTwarn, openHAB or Node-Red to translate to JSON.

Somebody did read the "Gotchas and Limitations" if subscribing via a client to write to the database?

I have another question. What is the purpose of posting MQTT from emonHub: https://github.com/emoncms/emoncms/blob/stable/docs/RaspberryPi/MQTT.md#...

Why should my graphical display of data do postings?

And just took a look what's happening in "input/view".
Oh my - no WebSockets to stream. Everytime a request...
I did not build a dashboard yet. Is that also the case there with life data?
HEYYYY we have Websockets for that. They are not new anymore ;-p

Regards from Germany from the guy who scrutinizes everything :-)