Submitted by fastbike (not verified) on Sun, 01/07/2012 - 10:38.
Seems to have been done for a few hours now
»
Re: Vis site down?
Submitted by TrystanLea on Sun, 01/07/2012 - 10:46.
O dear, sorry about this. I will look into it!
»
Re: Vis site down?
Submitted by TrystanLea on Sun, 01/07/2012 - 11:14.
Ok so for some reason the virtual machine had turned it self off.
I have restarted it and it now seems to be stuck just after starting apache2, hopefully it will get unstuck and complete soon. Will keep you updated.
»
Re: Vis site down?
Submitted by TrystanLea on Sun, 01/07/2012 - 11:40.
Ok its back on but the load is really high and there are a lot of concurrent apache connections.. not sure If its more than usuall or not.
»
Re: Vis site down?
Submitted by EnergyRnR on Sun, 01/07/2012 - 12:06.
Trystan,
let me know if you want me to get my Trial client to shut down the emonbase. I've just started uploading to the site, so let me know if it's an issue and I'll sort it. I'm going on vacation tomorrow but will be online during the week to check status. Do what you need to do if my connection is causing issues.... thanks, Eamonn
»
Re: Vis site down?
Submitted by Andygodber (not verified) on Sun, 01/07/2012 - 13:02.
Im just seeing "OK" after clicking the EMONCMS beta button....
»
Re: Vis site down?
Submitted by TrystanLea on Sun, 01/07/2012 - 14:40.
Ok its back up and running, although it does seem really slow. I dont think your one monitor Eamonn will be causing the problem.
It looks like the part of the problem may have been this:
However it does seem to be more too it than that. Part of the issue I think was due to cron.php taking a really long time to execute with the number of feeds now on the server bring really quite high, but its strange that it should suddenly affect it. I've disabled cron temporarily and so it at least seems to be running.
I will need to dig a little deeper to see if I can see why the slowdown.
»
Re: Vis site down?
Submitted by TrystanLea on Sun, 01/07/2012 - 14:42.
Im really sorry about the loss of data over the last 14 hours.
»
Re: Vis site down?
Submitted by Andygodber (not verified) on Sun, 01/07/2012 - 15:03.
Hmm - still flaky here ;)
»
Re: Vis site down?
Submitted by TrystanLea on Sun, 01/07/2012 - 18:26.
The problem seems to be due to too much mysql concurent processes. Everything operates fine when the number of processes in the
mysql> SHOW PROCESSLIST;
is below about 15.
However it can quikly escale up to 50 as it does quite often and even 150. When I then disable the api/post API disabling all feed storage and input processing, it takes quite some time for the process list to clear, upwards of a minuite. So it seems to be really struggling to get through the number of queries it is recieving.
Question is, have we hit a limit where before yesterday everything was running smoothly but now its really unsable and keeps getting stacked up with queries.
It might be possible to streamline the input processing code to minimize the number of queries somewhat but thats going to take some time.
Im trying to work out at the moment if its a particular input process that is causing the slow down, so thats why you might see some intermittent updating.
Thats where I've got to so far.
»
Re: Vis site down?
Submitted by bpbroom (not verified) on Mon, 02/07/2012 - 19:10.
Re: Vis site down?
Seems to have been done for a few hours now
Re: Vis site down?
O dear, sorry about this. I will look into it!
Re: Vis site down?
Ok so for some reason the virtual machine had turned it self off.
I have restarted it and it now seems to be stuck just after starting apache2, hopefully it will get unstuck and complete soon. Will keep you updated.
Re: Vis site down?
Ok its back on but the load is really high and there are a lot of concurrent apache connections.. not sure If its more than usuall or not.
Re: Vis site down?
Trystan,
let me know if you want me to get my Trial client to shut down the emonbase. I've just started uploading to the site, so let me know if it's an issue and I'll sort it. I'm going on vacation tomorrow but will be online during the week to check status. Do what you need to do if my connection is causing issues.... thanks, Eamonn
Re: Vis site down?
Im just seeing "OK" after clicking the EMONCMS beta button....
Re: Vis site down?
Ok its back up and running, although it does seem really slow. I dont think your one monitor Eamonn will be causing the problem.
It looks like the part of the problem may have been this:
https://forum.bytemark.co.uk/comments.php?DiscussionID=3736
However it does seem to be more too it than that. Part of the issue I think was due to cron.php taking a really long time to execute with the number of feeds now on the server bring really quite high, but its strange that it should suddenly affect it. I've disabled cron temporarily and so it at least seems to be running.
I will need to dig a little deeper to see if I can see why the slowdown.
Re: Vis site down?
Im really sorry about the loss of data over the last 14 hours.
Re: Vis site down?
Hmm - still flaky here ;)
Re: Vis site down?
The problem seems to be due to too much mysql concurent processes. Everything operates fine when the number of processes in the
mysql> SHOW PROCESSLIST;
is below about 15.
However it can quikly escale up to 50 as it does quite often and even 150. When I then disable the api/post API disabling all feed storage and input processing, it takes quite some time for the process list to clear, upwards of a minuite. So it seems to be really struggling to get through the number of queries it is recieving.
Question is, have we hit a limit where before yesterday everything was running smoothly but now its really unsable and keeps getting stacked up with queries.
Or is it stacking up because of this issue: https://forum.bytemark.co.uk/comments.php?DiscussionID=373
It might be possible to streamline the input processing code to minimize the number of queries somewhat but thats going to take some time.
Im trying to work out at the moment if its a particular input process that is causing the slow down, so thats why you might see some intermittent updating.
Thats where I've got to so far.
Re: Vis site down?
Hi Trystan. The server problems over the weekend might have been triggered by the leap second change. See http://www.theregister.co.uk/2012/07/02/leap_second_crashes_airlines/