I am having problems getting my NanodeRF to properly communicate with my personal Emoncms server. I have my EmonTX transmitting valid data to my NanodeRF, which is seen on the serial port of the Nanode. The Nanode is properly receiving an IP address when I try in both the DHCP and in the static IP cases. Using packet sniffing I have determined that the NanodeRF is sending out a valid URL that is reaching the server, but the server is responding with a 403 Forbidden error indicating that it does not have permission to access that part of the server.
I have verified that the server is setup properly by copying the URL stream from the serial monitor and pasting it in a browser and getting the 'OK' response from the server. Using a packet sniffer, the only difference I can see between packets from my browser and packets from my Nanode is in the Host: section of the http GET. In my browser, the HOST section of the packet lists the proper sending IP address, while the packet from the Nanode shows garbage after the Host: text. This is occuring whether I use DHCP to set the Nanode IP address or I set it manually. Any ideas on what to check?
Dan
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
In my browser, the HOST section of the packet lists the proper sending IP address
Sending IP address, or the IP address of the server? I think the HOST field is how apache determines which virtual host to send the request to.
I'm not particularly familiar with the Nanode, but from a quick browse, it looks like this field:
// 1) Set this to the domain name of your hosted emoncms - leave blank if posting to IP address
char website[] PROGMEM = "emoncms.org";
could be what's supposed to end up in the HOST field.
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Also, could you check in your server logs what page is giving the 403 errors ?
Your server logs should also log the complete request (since it is a GET request) with no need of the packet sniffer.
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Thanks for your assistance. I am using the NanodeRF_multinode_static_IP sketch (although I have tried and failed with all of the other sketches as well). Here are what I believe to be the relevant parts of the code. The API key has been hidden to protect the data. My local Emoncms server is 192.168.1.200 and my NanodeRF is the 192.168.1.201 address.
// ethernet interface mac address, must be unique on the LAN static byte mymac[] = { 0x42,0x31,0x42,0x21,0x30,0x31 }; // 1) Set this to the domain name of your hosted emoncms - leave blank if posting to IP address char website[] PROGMEM = ""; // or if your posting to a static IP server, change to true if you would like the sketch to post to static IP (not sure if this is working..) boolean use_hisip = true; static byte hisip[] = { 192,168,1,200}; //emoncms.org IP // Set to true and enter IP to give the NanodeRF a static IP address - default is DHCP (NB: must be used with static server IP (hisip) boolean use_staticIP = true; static byte myip[] = { 192,168,1,201 }; //NanodeRF static IP address - not needed if using DHCP static byte gwip[] = { 192,168,1,1 }; // gateway ip address - not needed if using DHCP // 2) If your emoncms install is in a subdirectory add details here i.e "/emoncms3" char basedir[] = "/emoncms"; // 3) Set to your account write apikey char apikey[] = "***my API Key***";My apache logs are not showing errors, possibly since the log level is not set properly. The access logs show the get request being received, but not an error. Here are the results of the packet sniffing, using Wireshark to follow the TCP data stream.
A GET request packet stream from the NanodeRF and the server response looks like this:
If I copy and paste the URL from the serial monitor, or from the packet stream and instead of sending it from the URL, send it from a browser I see this successful packet stream.
The two packets are different in a variety of ways, but my main concern is that the Host section following the GET request is blank on the failing packet. When trying this with the NanodeRF_multinode sketch (using DHCP to assign the NanodeRF an IP address), I was getting the same errors, but in that case the Host section had garbage in it instead of being blank. So, that leads me to believe that this part of the ethercard code is not working properly, but I cannot verify. I am not sure if this is something that everyone has, but my server configuration is different and therefore not tolerant of, or if it is a sketch error. To test that I will later today setup an account on the main Emoncms server across the pond and see if I can post to that server with this sketch or if I get the same error.
Apologies for the weird web site formatting. I can't seem to figure out to make it wrap the long lines. Thanks in advance for any assistance.
Dan
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
For what it's worth, I've only had success with a NanodeRF with a static dotted IP address (with WampServer under Windows). All your settings in the Nanode look sensible to me.
Is your server a separate machine, or is it running on the same machine as the browser?
When you sent the data from the browser, did emonCMS respond with "ok" (LOWER CASE, no quotes!) or do you mean Apache responded OK?
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Robert,
My Emoncms is running on a dedicated server, on my home network, under FreeBSD. I posted a tutorial on getting it running under FreeBSD awhile ago. I am using a laptop to do packet sniffing on the network, and looking at the stream between the NanodeRF and the dedicated server. When I copy and post the TCP stream from the packet sniffing and use the browser on the laptop, I am able to see the OK response from Emoncms, and the data shows up in the feeds. I can also do that manually from other computers on the network. While I waited for my EmonTX kit to arrive, I used a professional 3-phase power analyzer (real nice setup I borrowed from work) to log some data from my home/solar panels. I took the spreadsheet of logged data in Excel and wrote a macro to post it to the server every 5 seconds to simulate realtime data being posted. During that work I played with the dashboard and visualization features in Emoncms, so I feel confident that the Emoncms is working properly.
My Dad is in town and we are hooking up the EmonTX to the circuits today and doing the calibration using the power analyzer. So, I will try and use the hosted Emoncms in an hour or two once our calibrations are done. Thanks for everyone's assistance.
Dan
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Well, your server in Wales is receiving my data without error, so the problem is somewhere in my server configuration. I will start investigating that and determine what fixes it and post that here. If anyone has any ideas I would appreciate some hints since browser posted data is OK, but NanodeRF posted data is rejected. I will look at increasing the Apache logging and see if that points to anything.
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
On second look, packet sniffing the outgoing packets to Wales shows that the Host field of the packet is properly filled with "emoncms.org" so it may still be a NanodeRF issue. Anyone get the NanodeRF to work using a static IP address for the server?
Dan
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Does it help any if you load this guy:
char website[] PROGMEM = "192.168.1.200";
with the static IP address of your server as shown?
That might get your Nanode request looking more like your browser request in the HOST field.
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
As I said (maybe not too clearly), I have a NanodeRF working to Wampserver under Windows XP using static addressing. Here is the pertinent bit:
// ethernet interface mac address, must be unique on the LAN
static byte mymac[] = { 0x42,0x31,0x42,0x21,0x30,0x31 };
// 1) Set this to the domain name of your hosted emoncms - leave blank if posting to IP address
char website[] PROGMEM = "";
// or if your posting to a static IP server, change to true if you would like the sketch to post to static IP (not sure if this is working..)
boolean use_hisip = true;
//static byte hisip[] = { 213,138,101,177}; //emoncms.org IP
static byte hisip[] = { 192,168,1,65}; // IP of Machine with Wampserver running
// Set to true and enter IP to give the NanodeRF a static IP address - default is DHCP (NB: must be used with static server IP (hisip)
boolean use_staticIP = true;
static byte myip[] = { 192,168,1,66 }; //NanodeRF static IP address - not needed if using DHCP
static byte gwip[] = { 192,168,1,254 }; // gateway ip address - not needed if using DHCP
// 2) If your emoncms install is in a subdirectory add details here i.e "/emoncms3"
char basedir[] = "/emoncms5";
// 3) Set to your account write apikey
// char apikey[] = "----your API key ---";
char apikey[] = "b821f5936a56e44392c6f3bccf2bdcae"; // don't worry - this is a random string
//IP address of remote sever, only needed when posting to a server that has not got a dns domain name (staticIP e.g local server)
byte Ethernet::buffer[700];
That looks similar enough to me to think the problem isn't there - unless there's something funny going on with line endings or something like that.
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
My theory is (but it's nothing more than a hunch) that Robert's Wampserver is more forgiving about having a blank HOST field than Dan's apache is. From what I can tell (without have a setup here to experiment with), whatever you put in:
char website[] PROGMEM = "";
ends up in the HOST field. Since we know Dan's apache is happy when it sees 192.168.1.200 in the HOST field (as shown by his browser experiments), then perhaps forcing the Nanode to do the same may sort it.
Robert, do you have any way to inspect your packets to see if your HOST field is indeed blank with the above config?
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
It will be late tomorrow (15 hrs at best) before I can look at that - assuming my brain doesn't overheat. What I can add now is I've looked at the library used by the Nanode - and it's totally uncommented and impenetrable. I think - I'm not sure - that if it sees anything in PROGMEM, it uses that in preference to the IP address, even if you say don't.
I'm still running Apache of course, it's just that the configuration is likely to be not quite the same! I've got Wireshark so I'll get that out.
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Yeh, I think it very much depends on your virtual hosts config. There's a description of how apache uses the provided HOST: field here:
https://httpd.apache.org/docs/2.0/vhosts/details.html
... including which rule it follows when it's blank. So Dan, I suspect you've got two options:
1. find a way to force the Nanode to put the server IP address in the HOST field (like your browser does)
2. reconfigure your apache vhosts stuff so that blank or unknown HOSTs end up at the right virtual host
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Thank you all for the comments and information. I have tried in the past to put the IP address in the website field as dBC indicated above, but that did not work at the time. When I tried that I did not have packet sniffing going on, just seeing no response from the server and no data posted. My guess is that the code would try to do a DNS lookup on that IP and get a garbage result back.
Thanks for your extra words Robert. I have tried all of those items as well with no success. Static IP for NanodeRF with static IP for server, DHCP for NanodeRF (successful) with static IP for server, and all variations of the same. I have also tried using my dynamic DNS address that points to my WAN IP address, and turning on port forwarding to my server, so the packet leaves my house, bounces back, and then hits the server using name addressing. The DNS lookup correctly gets my external IP address, but the server still refuses it.
One challenge is that without packet sniffing, or digging into the Apache logs, you cannot tell the difference between the NanodeRF not reaching your server, and it getting refused for some other reason. The serial monitor says the same thing. It may be useful if the NanodeRF gets some form of response back to post that response code instead of only posting the Data Received message on a successful post.
Looking through the documentation on the Apache httpd.conf file, there is a spot where you list the server name, or the server IP address. I have edited that many ways as well without success. I am also attempting to get my router, which can act as a local DNS server, to give the Emoncms server a network name that I can try to address, but so far have not been successful at that. Anyhow, I will dig into the Vhosts link you have given later this weekend. I will be busy most of tomorrow so I won't get back to this until tomorrow night or Sunday. Thanks again for your collective thinking and hopefully when we figure it out we can get a root cause to share and help ease the collective burden. Good night.
Dan
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Hi, had a look at the ethercard code that the firmware is using .... If you have set up apache with a virtual host identified by a name, then you should put that name in the website array, and you should halso put the ip in the hisip array and set the boolean use_hisip to true
If on the other hand, your virtualhost is identified by an ip, you should put the ip, and still use the hisip in order not to trigger a dns resolution
If it is neither of these two, then you should post the relevant part of the apache config file (the virtualhost section) to see whether you are doing something weird with the folder access rules....
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Here is the NanodeRF successfully talking to Apache under Wampserver. I've given you the bytes for the whole frame so that you can see exactly what's being sent and returned.
I think this just confirms, Dan, that your Nanode is doing the same as mine. It looks to me as if Matt is on the right track...
(Falsified APIkey again!)
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
Yes, I believe it is a server name issue. I recall when working to get the system originally running I had two different configuration files where the server name was put in, but I can only find one right now. I will dig further in and report back when I figure it out. In the mean time, I am logging my data to the Wales hosted Emoncms server to do some power logging so I can see how accurate my calibrations on the EmonTX have been. I have limited use of a very nice Fluke 435 3 phase power analyzer from work so I am trying to compared summed values over time to see how accurate it runs compared to the Fluke. The generated readings are fluctuating quite a lot so I am interested in seeing if that noise integrates out. Once I see that accuracy over a few days I will try out the Emonlibpro that was recently posted to compare.
Thanks for your notes and information. I will update when I figure more out.
Dan
Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error
I have determined the problem and corrected it. The issue relates to addressing Emoncms servers only by IP address instead of by DNS name. The httpd.conf file used by the Apache server to set many server options has a section in which you tell it the server name. More information on it can be found here: http://httpd.apache.org/docs/2.2/configuring.html. The comments in the config file tell you if your server does not have a name, to put the IP address in it. But, even with that setting, it will look for the HOST: field of the GET packet to list that IP address before serving any files. Without that field being filled in, and nothing extra in the httpd.conf file to tell it what to do, it will return a HTTP 403 Forbidden code which the NanodeRF does not know what to do with.
By adding a virtual hosts section to the httpd.conf file, a server can serve multiple sites from one IP address, matching the hosts field with the server name for each host (http://httpd.apache.org/docs/current/vhosts/examples.html). In this case, we can use the virtual hosts section with a wildcard "*" to tell it what to do with packets that have the HOST field blank. So, adding:
to the httpd.conf file right after the default document root section of the config file fixed it. The DocumentRoot is simply the default root for the apache server as a whole. The *:80 in the VirtualHost header tells it what to do if a packet arrives that does not match any other listed virtual hosts. Since I don't have any other hosts listed, it defaults to the root for all packets.
So, to summarize, if you want a static IP for your personal Emoncms server, and you are getting 403 forbidden errors (which the NanodeRF will not report but you may see in the Apache logs if you set it to list the details (http://www.cyberciti.biz/faq/apache-logs/), set the ServerName to your static IP and add the above lines below the DocumentRoot section, and it should work.
Let me know if you have any questions. Thanks to all who assisted and for my Dad who tracked down the page on virtual host settings.
Dan