Remove spike from feed

Hi all,

Is there a script to be able to remove spike from data collected ? since I can remove data point for mysql feed using my sql query (which was also working from gui using vis edit realtime data) but for phpfiwa I did not suceeded to remove any data point (even from the GUI)

Ideally a little script called with feed number and date/time with new data value would really be fine, or even just to delete the bad one without assign a new value.

I know reading some other posts that I'm not the only one with this need, when developing some custom node  to collect data sometime during test the data we send to emoncms is bad (due to testing mode of our work) and the possibility to remove it from graph and storage would be very helpfull even if it is some script and not in the GUI. If the feature is implemented to manipulate some data into phpfiwa we could also then do some scripting to remove spike automatically each day for example

Any other method will also be fine, it's just an idea.

Thanks everyone for your help.

 

 

TrystanLea's picture

Re: Remove spike from feed

Hello Charly86

I wrote a script for this just a couple of hours ago for use with phpfiwa feeds. It works through the phpfiwa data files searching for values that are larger or smaller than the limits, when it find one it sets it to NAN which means its ignored in the graph.

If you copy the following into a file called say: removespikes.php. Then run with sudo php removespikes.php it should do what you want. I would recommend making a backup before you run it just in case.

 

<?php

$dir = "/var/lib/phpfiwa/";

$id = 0;

for ($n=0; $n<4; $n++)
{
    if (file_exists($dir.$id."_$n.dat"))
    {
        $npoints = floor(filesize($dir.$id."_$n.dat") / 4.0);
        $fh = fopen($dir.$id."_$n.dat","c+");

        for ($i=0; $i<$npoints; $i++)
        {
            $tmp = unpack("f",fread($fh,4));
            $val = $tmp[1];
           
            if (!is_nan($val))
            {
                if ($val>1000000)
                {
                    echo $val."\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

                if ($val<-1000000)
                {
                    echo $val."\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }
            }   
        }
       
        fclose($fh);
    }
}

 

 

Charly86's picture

Re: Remove spike from feed

Tristan,

thanks for your quick reply, just tweaked for the values I needed to remove, this is working fine on the first time.

Great job, I think this script could be placed into useful scripts folder, because it is really a excellent one, I will try to change it to have set of parameters to be as kind as "universal" spike remover.

I will consider also to have this script to try to analyze spikes (looking values before and after a point) as soon as I will have some time.

Thanks again for your great job

Charles

 

Eric_AMANN's picture

Re: Remove spike from feed

Hi,

Very interesting script until the edit-realtime feature is not back.

It would be great to be able to run this script with our own feeds stored on emoncms.org.

One should also be able to spikes in histgraph.

Eric

 

 

Charly86's picture

Re: Remove spike from feed

Ok guys, I changed the script to be able to get values from command line, assuming you named the script remove_spike.php the launch command is now

sudo php remove_spike.php -i FeedId -n MinValue -x MaxValue

for example to remove value above 5.5V and below 1V (battery feed in V for example) on feed 22 the syntax is

sudo php remove_spike.php -i 22 -n 1 -x 5.5

Here is the new script

<?php

$dir = "/var/lib/phpfiwa/";

function usage($scriptname)
{
  print $scriptname ." usage: \n";
  print $scriptname ." -i feedid -n minvalue -x maxvalue\n";
}

// arguments parameters
$opts = getopt("i:n:x:");

// Handle command line arguments
foreach (array_keys($opts) as $opt)
  switch ($opt)
  {
    // feed id
    case 'i': $id = $opts['i'];  break;
    // min value
    case 'n': $min = $opts['n']; break;
    // max value
    case 'x': $max = $opts['x']; break;
  }

// check all parameters are good
if (isset($id) && isset($max) && isset($min))
{
  print "Feed ID   : ".$id  ."\n";
  print "Min value : ".$min ."\n";
  print "Max value : ".$max ."\n";
}
// Error display usage
else
{
  usage($argv[0]);
  exit(1);
}

for ($n=0; $n<4; $n++)
{
    if (file_exists($dir.$id."_$n.dat"))
    {
        $npoints = floor(filesize($dir.$id."_$n.dat") / 4.0);
        $fh = fopen($dir.$id."_$n.dat","c+");

        for ($i=0; $i<$npoints; $i++)
        {
            $tmp = unpack("f",fread($fh,4));
            $val = $tmp[1];

            if (!is_nan($val))
            {
                if ($val>$max)
                {
                    echo $val." > ".$max ."\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

                if ($val<$min)
                {
                    echo $val." < ".$min . "\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

            }
        }

        fclose($fh);
    }
}

 

Ian Eagland's picture

Re: Remove spike from feed

Hi

Is there a version of this script to correct phptimeseries which have to be used for daily data?

I have ended up with a negative entry due to an emontx reboot.

http://79.78.250.151/emoncms/ianeagland&id=4

Regards

Ian

Paul Reed's picture

Re: Remove spike from feed

That's a handy little script to remove data glitches, such as the errors made whilst setting up emoncms inputs/feeds, and works without issue.

Charles, your further development of Trystan's script makes it a very flexible tool, and I agree, it would be a great addition to the useful scripts Git folder.

Paul

Edit - I've wrtten it up and submitted a git pull for consideration

TrystanLea's picture

Re: Remove spike from feed

Bra1n's picture

Re: Remove spike from feed

I've attempted to edit the script to use with phpfina but although it appears to work the values come back again, I'm guessing there's some other storage location and/or process going on which reinserts the data, my modified version below

<?php

$dir = "/var/lib/phpfina/";

function usage($scriptname)
{
  print $scriptname ." usage: \n";
  print $scriptname ." -i feedid -n minvalue -x maxvalue\n";
}

// arguments parameters
$opts = getopt("i:n:x:");

// Handle command line arguments
foreach (array_keys($opts) as $opt)
  switch ($opt)
  {
    // feed id
    case 'i': $id = $opts['i'];  break;
    // min value
    case 'n': $min = $opts['n']; break;
    // max value
    case 'x': $max = $opts['x']; break;
  }

// check all parameters are good
if (isset($id) && isset($max) && isset($min))
{
  print "Feed ID   : ".$id  ."\n";
  print "Min value : ".$min ."\n";
  print "Max value : ".$max ."\n";
}
// Error display usage
else
{
  usage($argv[0]);
  exit(1);
}

for ($n=0; $n<4; $n++)
{
    if (file_exists($dir.$id.".dat"))
    {
        $npoints = floor(filesize($dir.$id.".dat") / 4.0);
        $fh = fopen($dir.$id.".dat","c+");

        for ($i=0; $i<$npoints; $i++)
        {
            $tmp = unpack("f",fread($fh,4));
            $val = $tmp[1];

            if (!is_nan($val))
            {
                if ($val>$max)
                {
                    echo $val." > ".$max ."\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

                if ($val<$min)
                {
                    echo $val." < ".$min . "\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

            }
        }

        fclose($fh);
    }
}

gluino's picture

Re: Remove spike from feed

Hi all,

How can I deal with a spike on hosted emoncms.org?

I just got a huge spike in my temperature/humidity data, that is squishing the good data due to the chart auto-scaling. Alternatively, if only there was a way to manually set the chart vertical scale, then I can leave the glitches alone. This is the first huge spike after charting about 15 units of EmonTH's for about half a year.

JBlackham's picture

Re: Remove spike from feed

Stuck in same situation.

andrea.rimo's picture

Re: Remove spike from feed

I have the same issue... spikes on the battery voltage reading are messing up the scale of the graph making it impossible to see small variation (the spike is 600V for 2 AA battery it's just flattening the whole plot)

I tried the EditRealTime but it doesn't work. there are posts that it has been disabled because of a bug.

Any news on this?

 

Jon's picture

Re: Remove spike from feed

I have a similar issue with EditRealTime.  I tried to delete a spike from July 15, 2015 at 2:54:50 PM CDT and it did not work.  When I looked at the graph, the EditRealTime ended up changing a value on March 18, 2016 at 10:21:59 PM CDT.  This is the time I was doing the editing.  (Now I have two spikes to fix)

emonPi image emonSD-20Feb16 and low-write 9.4 | 2016.03.15

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.