Archived Forum

You'll need to tell us more about what you want to do with the data, if using a data stick to pass from one machine to another you may get gaps in the data.etc.

The pre-loaded image includes a server and emoncms so if you attach a monitor and keyboard, which I guess you must if you have no network. You can just open a browser on the Pi and the "localhost/emoncms" website will be available on that Pi so you can create dashboards and graphs etc and the data will be coming in live from the emonTx.

Paul 

Hi Paul 

Many thanks for your response, I want to use the logged power data (which is the school power profile) to design how big PV panels and energy storage for the school are required and for the research study. The equipment will be installed at the school as a standalone equipment and the data will be logged into the USB stick in the form of a text file or a CSV file. Every one or two months I will go to a school to collect the data from the USB stick. I have all software ( from openenergymonitors) downloaded into emonTxV3 and raspberry pi2 and it works well and shows something on the emoncms website. I just need the raspberry pi2 to send the data to the USB stick and I do not know how to modify the exist code at all. Do you have know how to add any commands or sort codes so that the raspberry pi can write the logged data into the USB stick at all? (it does not matter that much about the delay of the data sending)

Many thanks

Seksak

Duplicate post deleted - Moderator (RW)

What is your timescale ? 

There is no current capability for this but I have in the past had a "csv reporter" functioning with emonHub that would suit your needs, emonHub has changed since it was last used so I will need to locate, revive it and make some changes.

Paul

Hi Paul 

I am planning to install the equipment by the end of August so if you could help me with this, I would appreciate it. 

Many thanks

Seksak

I'll see what we can do. No promises, but I'm sure we can knock something up, if you are not too fussy :-)

It will however, be for "original" emonHub, can you confirm which version you have? I think the best option would be to just have emonhub on a read-only OS, simpler to set up, less going on and better suited to a remote un-supervised install.

Paul

Hi Paul

I have emonTX V3 433MHz and raspberry pi2. The programme which was downloaded into the raspberry pi2 is emonSD-17Jun2015. 

Seksak

Hi,

I've added a class to emonhub that writes csv files with data received from emontx and a emonlcd.

It is compatible with current emonhub and ads writing file capability plus interface with arduino that receives data from old oregon scientific sensors I had home

It produces one file per day and per node.

If you are interested I can share the code.

Hi 

Certainly, I am very interested in that. Could you please share the code with me? Does the code run on raspberry Pi?

Many thanks

This is the code I use, it is very rough and can be much nicer, but just in case it is useful for someone:

Append to emonhub_reporter.py:

(/home/pi/emonhub/src/emonhub_reporter.py)

class USBReporter(EmonHubReporter):

    def __init__(self, reporterName, queue, **kwargs):
        """Initialize reporter

        """

        # Initialization
        super(USBReporter, self).__init__(reporterName, queue, **kwargs)

        # add or alter any default settings for this reporter
        self._settings.update({'file': ''})

        # This line will stop the default values printing to logfile at start-up
        self._settings.update(self._defaults)

        # set an absolute upper limit for number of items to process per post
        self._item_limit = 250

    def _process_post(self, databuffer):
        """Save data in csv file."""
        
        # databuffer is of format:
        # [[timestamp, nodeid, datavalues][timestamp, nodeid, datavalues]]
        # [[1399980731, 10, 150, 250 ...]]
        
        try:
            f = open(self._settings['file'], 'a')  #open file for appending
            writer = csv.writer(f)
            writer.writerows( databuffer )
            return True
        except IOError,v:
            self._log.warning("IO error upon writing to USB drive: "+self._settings['file']+'error code'+str(v[0])+str(v[1]))
        finally:
            try:
                f.close()
            except:
                pass

       

And in the config (/boot/emonhub.conf)

[reporters]

# This reporter sends data to emonCMS
#[[emonCMS]]

#etc

#usb logging
[[usb]]
        Type = USBReporter
        [[[init_settings]]]
        [[[runtimesettings]]]
        file = /media/usb/data.csv

 

My solution was more complicated, I think this one is better, I would just add a date in the file name to avoid files to big to handle, instead of

f = open(self._settings['file'], 'a')  #open file for appending

use:

f = open(
date.today().strftime("%y-%m-%d") + '_' +
self._settings['file'] + , 'a')  #open file for appending

That's a good idea for certain situations!

I forgot to mention, you also need to import the csv module. So at the top of the emonhub_reporters.py:

#csv module for USB LOGGING
import csv

By the way, I use the word 'USB' a lot in the code, of course this code will work for any location not only your USB drive. So I should have called the reporter a CSVreporter instead.

To check if it is working you can do a:

tail -f /media/usb/data.csv

in your command line (replace 'data.csv' with the correct filename), you should see the data being added to the csv file.

If it is not working (possibly the emonhub daemon has no access to the file) you can check the emonhub log with this command line command:

tail -f /var/log/emonhub/emonhub.log

Hi 

Thank you for your help. But I need some more instruction for how to do it then. So I have tried as:

1. I do not actually have the  emonhub_reporter.py file in the folder and therefore I have created it and put all the code on and save the file as  emonhub_reporter.py at /home/pi/emonhub/src/.

2. Then I have added the code [reporters] into the config file which is emonhub.conf.

Is this the right way to do? How to test the code is working? Do I need to type any command on LXTerminal to activate the code at all?

(I have done as 1 and 2 but there is no csv file on the USB stick at all)

Please advise

Many thanks 

Apparently the source code has moved.. you need to find where the original file is located and then replace that with you modified code (after making a backup copy of the original).

Try to issue the command

locate emonhub_reporter

in your LXTerminal to locate the original file.

Using a custom reporter is how I have done it in the past, the concept is pretty much the same. I didn't use the csv module however so I will take a look at that.

Interesting point about the log file naming, I haven't used this for a prolonged time so hadn't experienced a problem with file size, but it is true that even the 5mb logfiles are cumbersome to handle on a Pi. do you then need to open each file or do you have a way of opening and concatenating the contents of multiple files in one go? 

Something else to consider is writing to disc in blocks of data to prevent SDcard (or usb stick) failure, due to the small data size the packets get rewritten several times until that on-disc-block is full, for example by writing on every 10th frame increases disc life expectancy ~10x,

@Seksak - These examples are for original emonHub not the emon-pi variant, you will need to switch to another image or create your own Raspberry Pi 2 pre-built SD card image (emonSD-13-03-15.img)  ( also see this thread for some links ) as the emon-pi variant doesn't have reporters and the imports/threading/messaging is all very different.

Paul

Interesting thoughts about the blocks of data being rewritten each time. The block size is probably 512 bytes isn't it?

What is your implementation of this 10th frame solution?

I guess something like this might work:

        # add or alter any default settings for this reporter
        self._settings.update({'interval':10, 'batchsize': 100, 'file': ''})

In case your data interval is 1 second, this will result in 10 datapoints written at once I guess?

I am not sure about the batchsize, but after quickly scanning through the code it seemed like it defaults at 1 and is a bottleneck for the number of datapoints that is saved at once. The EmonCMS reporter also has batchsize at 100.

You could even increase the interval to make sure it is writing a block max two times in the worst case scenario.

The interval and batchsize  in the emoncms reporter are used to regulate the traffic to server, the interval defines the minimum interval between sends and the batchsize then limits the number of frames in each send as a network outage could result in a backlog of 1000's of frames buffered, originally implemented in case rmoncms.org was down for maintenance etc when it came back up there was some traffic calming.

However the variables are defined in the root reporter/interfacer so they can be used or redefined in each new class, I was thinking interval could be the max time to go between saves, just to persist any data held in ram for a prolonged time due to slow updates and batchsize could define the min (target) sizr for saving, This can easily use the existing frame count method, from which a user can calculate from frame size/frequency how often to save or it could perhaps be changed to calculate the actual size on disc for each save.

I don't think it would be too difficult but, I also don't think it is worth being too accurate, unfortunately even if we did know how to determine the block size of each SDcard, we do not know where the blocks start so even if we matched the size exactly, it would hit 2 blocks.But that would be the most efficient scenario I guess.

The close method would also need to save any outstanding packets.

Paul

By coincidence I was working on the USB logger again because I added a second USB stick to my setup for redundant logging. I noticed that when the USB stick is unplugged the reporter keeps trying to write to it, without delay.

Looking at the source code I concluded that the interval setting is only taken into account for successfull attemps:

if self._process_post(databuffer):
                [....]
                # log the time of last succesful post
                self._interval_timestamp = time.time()

Which means that when the USB drive is not connected, or emonCMS is down, the reporters will retry constantly without pause. This may not be a problem for the USB stick, but might be worth considering for emonCMS.

Yes the emoncms interfacer could probably have some throttling added, although there is actually a 60sec timeout on each request attempt so it is only if there is no connection that the retry frequency increases at which time it shouldn't cause emoncms an issue.

The CSV interfacer would also benefit from slowing the retries and reduced error logging. in v2.0 I had experimented with a retry 5 times quickly which would then pause for a time before retrying at a regular interval pattern.

Paul