How to look at your energy consumption on the web
This tutorial details how to first connect your energy monitor to the web with an ethernet shield and then how to set up a database on a web-server for the ethernet shield to connect to and then to finish how to read the data from the database and produce a nice zoom-able graph using flot.
I'd like to thank John from jarv.org for his tutorial and help on logging using the ethernet board although he is using open flash charts and sql lite instead of mysql and flot much of the implementation is very similar.
Part One - The Ethernet shield
The first step is to get an Ethernet shield for the Arduino. There are two Ethernet shields available to my knowledge:
- The official ethernet shield from Arduino themselves. Costs £29.00 from cool components.
- Nuelectronics ethernet shield which is a bit cheaper. Costs £12.50 from nuelectronics.
The arduino sketch below works with the official ethernet board library. There is a guide to do the same thing but with the nuelectronics board here: Nuelectronics web client example
Once you have your ethernet shield the next step is to connect it up to an Arduino and upload the sketch.
Arduino Sketch
Note: This sketch is just sends set values out to the server, its just to show the basics needed. Have a look at the whole house energy monitor documentation page for an example of how this can be connected up to the rest of a setup.
Download: SAethernet.tar.gz
There are a couple of changes that need to be made to the arduino sketch now so that we are ready for the next section.
- Make sure the MAC address and ip of the ethernet board are unique on your local area network
- Enter the ip of your webserver. If you have a shared hosting you can find the shared ip using a trace-route program, you can trace-route from network tools in ubuntu.
- If you have a shared hosting you will also need to change the "GET /cgi-bin/post.pl?L=" to include your server url: "GET yoururl.org/cgi-bin/post.pl?L="
The next step is to upload the sketch to the Arduino and then to set up the database.
Part Two - Setting up a database on a web-server and then connecting to it.
The first thing you are going to need is a web hosting. If you dont already have a web hosting you can:
- Get a web hosting, Im with WebHost.Uk
- Setup a LAMP server on a spare computer running ubuntu here's a tutorial
- Use Pachube and skip this tutorial (check out this one out in stead) , this is the simplest, the service has been designed with this kind of application in mind. However at the moment from what I can tell you cant zoom and look about the graphs in any detail, but maybe this will change in the near future.
Once you have your server whether its on your local network or some ware in a far away place the next step is to set up a database to place the data we want to graph in.
Create a database with Cpanel (skip to phpmyadmin step if you don't have cpanel)
- Go to MySQL Databases
- Enter a name for your database, for example "power". Click on create database.
- Add a user to your database, create a user with minimal privileges SELECT and INSERT.
- Note down your database name, username and password you will need them again soon.
Setup database with phpMyAdmin
- Open phpmyadmin.
- Select your database from the list on the left (or create one if you didn't have cpanel to do the steps above.)
- Create a new table called DATA to with 2 fields A and B.
- On the next page, lets make the field A the field that holds the Power data and lets make the second field B hold the time. Make the power field an int(11) and the time field a bigint(20). Set Null to no and Default to None. Click Save and then Go.
- That's the database ready! The next step is to setup a perl script to access and insert values into the database.
Setting up a perl script
The perl script can be downloaded here: post.tar.gz
- Enter your database name and username in the post.pl file.
- If your using cpanel: Open file manager and locate the document root of your domain. If you have a server on a local computer locate the folder /var/www.
- Create a folder named "cgi-bin" if it doesnt exist.
- Upload the perl scrip to the folder and set its permission to 755.
- Ok we are ready to test it
- Enter this in your address bar: "http://yoursite.org/cgi-bin/post.pl?L=100&C=200"
- Now check your database in phpMyAdmin, click on Browse, hopefully you should see the values 100 and 200.
Start up the Arduino
If all is well, when you connect up the Arduino you should start seeing values being entered into your database.
Adding some basic security
At the moment anyone can enter the above address in to their address bar and add values to your database. You can make it possible for only you (your ip address) to have access to the post.pl file by creating a .htaccess file in the cgi-bin directory with the following content:
--------------------------------------------------------
AuthGroupFile /dev/null
AuthName "cgi access"
AuthType Basic
<limit GET>
order deny,allow
deny from all
allow from --your ip address--
</limit>
--------------------------------------------------------
If you don't know your ip address: http://www.whatismyip.com/
Part Three - Reading from the database and graphing with Flot
Screenshot of flot based graph showing the immersion heater going on
- Download Flot here.
- Upload Flot to the document root of your domain and unzip.
- Download the flot configuration file/page Graph.tar.gz this is the web page that accesses the database and creates a flot graph from it.
- Upload to the document root of your domain and unzip both index.php and layout.css to the document root.
- Enter your database name, username and password in index.php.
- Set permissions to 0644 on both files.
Great that's it! Have a look at your domain, the graph should appear! and look something like this.
| Attachment | Size |
|---|---|
| Graph.tar.gz | 1.49 KB |
| SAethernet.tar.gz | 1000 bytes |
| post.tar.gz | 681 bytes |
Re: How to look at your energy consumption on the web
Hey,
i like what you told all about this. all the steps are told by you in a simple manner. i will also try myself to connect monitor to the web with an ethernet shield by following your steps.
Re: How to look at your energy consumption on the web
I am using the Ethernet shield. and Im not sure what these stands for:
byte ip
byte mac
byte gateway
byte server
if i understood the byte ip, is the ip address that I have from my internet supplier... the byte mac is the number which comes with the ethernet shield... but the gateway and the server once i cant understand.. can you help me pls
Re: How to look at your energy consumption on the web
IP - is the IP address of the Arduino, make sure its an unique ip on your network (usually something like 192.168.x.xx)
mac - is the MAC address of the Ardunio again make sure its unique on your network. Unless you have more than one Arduino connected to your network you probably done need to change this.
Gateway - The IP address of your internet router
server - The IP address of the server you wish to send data
If you are using windows you can find out the gate way IP and the IP of your computer by typing ipconfig into a cmd window. If using linux the terminal command is ifconfig.
Hope this helps, Glyn.
Re: How to look at your energy consumption on the web
mac: the MAC (Media access control) address for the device (array of 6 bytes). this is the Ethernet hardware address of your shield. Newer Arduino Ethernet Shields include a sticker with the device's MAC address. For older shields, choose your own.
ip: the IP address of the device (array of 4 bytes)
gateway: the IP address of the network gateway (array of 4 bytes). optional: defaults to the device IP address with the last octet set to 1
subnet: the subnet mask of the network (array of 4 bytes). optional: defaults to 255.255.255.0
i found this on the arduino site... so for the mac i just the number on the sticker found on the shield...the others ok... but hmm the server is to the website im going to built?
Re: How to look at your energy consumption on the web
the ethernet shield is connected directly on the arduino which is used in the measurement board?? or it has an arduino seperate for its own, and therefore the measurement board and the ethernet shield are connected with the XBEE??
Re: How to look at your energy consumption on the web
Hi,
Either is the answer. You can use one Arduino to take the measurements and post data online using the etherent shield or if an ethernet connection is not available near where you want to take the measurements you could use an Xbee or simple of the transmit the data to another Ardunio + etherent which would then post online. My advice would be to start with the former, its simpler!
Glyn.
Re: How to look at your energy consumption on the web
many thanks
Re: How to look at your energy consumption on the web
Hello Trystan. I began trying to implement your open energy monitor project some months ago, but my efforts have been discontinued. Now I am back, and hope will finish it soon.
I have setup a -free for the moment- web hosting, where I have created a mysql db, plus a php page to upload values, another one to query the values, and also, I have upload your flot environment, where I can see the results.
My arduino + ethernet shield work correctly. I can for example serially print the results of a GET request of, for example, a search in google. I can do the same thing with your http://openenerymonitor.org/emon URL.
Although I can manually and successfully upload values to my database, issuing for example a http://mieconsumo.260mb.com/kk.php?watts=247 request from my web browser, I can not do the same from the arduino + ethernet shield (some server in the hosting domain prevents me from that). Can you give me a hint what is going on? When I ping the mieconsumo.260mb.com url, I obtain an IP address, but the packages are not returning, giving timeouts.
Thanks in advance, you are succeeding in expanding knowledge throughout the world, and that I assume is a big satisfaction for you.
Angel Valor
Re: How to look at your energy consumption on the web
I am very glad to say that I have solved the issue... making two additions to the request that is sent to the server. First, as recommended in different arduino forums, adding the "Host:" http header to indicate my particular site to the shared hosting. This being necessary, was not enough to get the stuff working... I realized the problem could be connected to some security enforced in the hosting, that serves pages only to recognized web browsers such as IE or Mozilla. So I added the "User-Agent:" header to the request and it worked!!!
This is the code. Thanks. Regards, Angel Valor
if (client.connect()) {
Serial.println("connected");
client.print("GET /kk.php?watts=4567");
client.println(" HTTP/1.1");
client.println("Host: mieconsumo.260mb.com");
client.println("User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-MX; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB6");
client.println();
delay(1000);
} else {
Serial.println("connection failed");
}
}
Re: How to look at your energy consumption on the web
Glad you got it to work! and thanks for the finds! I saw people adding in host and user-agent and found I did not need them for posting to the two servers I use.. interesting.. If I get the problem in the future I will now know what to try first, thanks!