Feeds export feature: what's the timedate format in CSV?

Hi everyone, I've been trying this (new to me) feature to backup my data and I got CSV files.

That's how LibreOffice opens it:

…
1388666100 357.90
1388666400 238.60
1388666700 863.67
1388667000 2 071.00
1388667300 1 763.10
1388667600 373.80
1388667900 281.38
1388668200 196.63
1388668500 202.20
1388668800 195.00
1388669100 300.20
1388669400 318.10
…

The problem is how to decode (interpret) the first value: the date!

It looks like it is a seconds counter but… what's the starting date?

Dividing it by 60 (seconds) and 60 (minutes) and 24 (hours) I got a valid date but in 1943! :D

A second problem is that sometimes there is an unknown value on the second field which shifts the power (in this case) to third… Does anybody know what is?

Thanks in advance.

pb66's picture

Re: Feeds export feature: what's the timedate format in CSV?

Unix time counting starts 1/1/1970 not 1900. So in excel, I use the formula =(((A1/60)/60)/24)+DATE(1970,1,1).

Regards the "3rd value", I think a thousands separating comma is being picked up as a CSV separator and those values should be 2071 and 1763.1. That would also explain the leading zero on the "071.00" value. If that is the case, try and find where the thousands separator is being introduced and change it there, or, as a work around for this example, concatenate the 2nd and 3rd values.

Paul

gippy's picture

Re: Feeds export feature: what's the timedate format in CSV?

Thank You so much!

It was really simple then… sorry for the annoyance.

It works!

gippy's picture

Re: Feeds export feature: what's the timedate format in CSV?

Anyway I think that the behavior about thousands should be corrected, since a Comma Separated Values files should have commas only to separate values, or at least should have those values delimited also by ".

Or could be used a different separator such as Tab…

But the best, to me, would be avoiding thousands marking.

Thanks again, have a nice Sunday!

mircsicz's picture

Re: Feeds export feature: what's the timedate format in CSV?

If you don't like office you can use this shell script:

#!/bin/bash
INPUT=$1

#echo ${INPUT##*/}
OUTPUT=${INPUT%.*}
OUTPUT=$(echo "$OUTPUT"_converted.csv)

echo "TIME, VALUE" > $OUTPUT
conversion () {
while read LINE
do
        VALUE=$(echo $LINE|cut -d',' -f2)
        UTIME=$(echo $LINE|cut -d',' -f1)
        TIME=$(date -d @$UTIME +'%y-%m-%d %H:%M:%S')
        echo "$TIME, $VALUE" >> $OUTPUT
done < $INPUT
}

usage () {
                echo "Usage: emon_csv-conv.sh filename-to-convert.csv"
                echo ""
                echo "Use this shell script to convert CSV's exported from emonCMS to humanreada$
}

if [ $# -ne 1 ]
then
    usage
    exit 1
else
case "$INPUT" in
*.csv | *.CSV )
        conversion
        ;;
*)
        echo "it's no CSV"
        ;;
esac
fi

Greetz
Mircsicz
craigfryer's picture

Re: Feeds export feature: what's the timedate format in CSV?

The time system used by EMONCMS is the Linux time system, which starts at 00:00 hours on 1/1/1970.

Thus in MS Excel you can convert the time using the following:
=(CELL+TIMEZONE_OFFSET)/(60*60*24)+"1/1/1970"
Where:
CELL contains the Linux time value
TIMEZONE_OFFSET is used if you wish to state the value in your local time zone. If you are at located in the GMT +10 hours, then you would enter (60x60x10) or 36,000.

I suggest having two columns containing this formula, then in MS Excel format one cell as date and the other as time. Alternatively use a custom cell format.

craigfryer's picture

Re: Feeds export feature: what's the timedate format in CSV?

The easiest work around for the thousands separator comma in the value is to rename the file with the txt extension. Then in MS Excel open the file and select fixed width. Then create a column break before the first comma and another comma after the first comma. This will create three columns when the data is imported: time value, a comma and the value. Then you can simply delete the comma column. Thus no need to use search and replace.

Of course this won't work if the time set happens to include a value where number of digits in the time value increases.

Scott Geller's picture

Re: Feeds export feature: what's the timedate format in CSV?

Thanks for this thread - directly answered my question.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.