Home energy data logging project

strider3700 · 12-19-10, 05:10 PM

Well 6 days later everything is still working although I have found a limitation to my current logging technique. Saving a row every minute in a CSV file works just fine. The file is now 323.1kb which works out to about 20 MB a year which is nothing in my mind. It is however 8600 or so rows and open office has started to gak on it. Opening the file now takes 20 seconds and creating a chart takes close to 2 minutes. If I try to do any smoothing of the data it takes over 10 minutes. Simply resizing the chart when it's produced grinds on the new very fast quad core 8 gig's of ram machine.

This isn't shocking there is a reason real databases don't store everything in csv files. Things are really busy here but I may need to start on the link to the google apps DB or a local DB really soon. I want to learn the google apps version and get that data up for everyone to see but local is free, easy and not too difficult.

Another thing I also need to do is take a look at how much space is on the arduino it's self and see if I can get the code to store the data for awhile if the serial port isn't available. I had to reboot once and lost 4 minutes of data, which is far from critical but I like my data to be complete.

Thread Tools
Show Printable Version Email this Page
Display Modes
Switch to Linear Mode Switch to Hybrid Mode Threaded Mode

12-19-10, 05:10 PM	#16
strider3700 Master EcoRenovator Join Date: Dec 2008 Location: Vancouver Island BC Posts: 745 Thanks: 23 Thanked 37 Times in 30 Posts	Well 6 days later everything is still working although I have found a limitation to my current logging technique. Saving a row every minute in a CSV file works just fine. The file is now 323.1kb which works out to about 20 MB a year which is nothing in my mind. It is however 8600 or so rows and open office has started to gak on it. Opening the file now takes 20 seconds and creating a chart takes close to 2 minutes. If I try to do any smoothing of the data it takes over 10 minutes. Simply resizing the chart when it's produced grinds on the new very fast quad core 8 gig's of ram machine. This isn't shocking there is a reason real databases don't store everything in csv files. Things are really busy here but I may need to start on the link to the google apps DB or a local DB really soon. I want to learn the google apps version and get that data up for everyone to see but local is free, easy and not too difficult. Another thing I also need to do is take a look at how much space is on the arduino it's self and see if I can get the code to store the data for awhile if the serial port isn't available. I had to reboot once and lost 4 minutes of data, which is far from critical but I like my data to be complete.