Home energy data logging project

[Daox]

Have you looked into how much onboard memory there is on the arduino and how much time worth of data you can store? I know the newer chips have more memory on them. I'm not sure what one you are using.

[strider3700]

It's the at mega with the 1280 chip so 128K of memory for the programs. It's only got 4kb of eeprom that I can read and write to but if I use it efficently thats a bit. My daily csv files come in at 65ish kb without any attempt to minimize their size (It's storing 10 or so decimal points on temperatures....)

I want to lookinto some external memory for it though. I saw some 1 wire memory chips when I was getting samples from dallas I will track them down again and see what they offer or if others have gone a different route.

Hmm I just found that there is a library to use the onboard flash for storage. Since my current programs use 8k of the 128k I can probably get away with using that for a days storage easily enough.

So far I have to say I'm loving this project.

[strider3700]

On a totally different note the mail man just dropped off a box of 8 current sensors from Bourns inductive products. I sort of remember requesting these as samples 2 or 3 months ago. They say 7203-RC on them. I'll have to look up the data sheet later. I'm more then willing to try and promote any product that manufacturers send me free samples of.

If I buy the product and it's garbage though Then my online slamming knows no bounds.

[strider3700]

I stand corrected. The flash memory on the arduino appears to be not applicable for the storage of data like this. So it's back to an external chip or the 4kb eeprom. Joy.

[Daox]

Are you sure? I could have sworn it was... Why can't it be used?

[strider3700]

well it appears that you can write to flash however it's only meant to be written to 10,000 or so times. Sounds like a lot but I currently write 1440 times a day so there is a good possibility I'd be pushing it's life expectancy within a month.

It also appears to be a glorious hack of a method to make it work.

apparently I have 8K of ram to work with and since I have an external power plug and a 9V battery I may just check to see how much space in ram I actually have available and store in there.

I guess the first step is to sit down and figure out just how small I can get the data before storage. Some things can be greatly simplified. my current regular row of data looks like this.

2010/12/30 00:55 17.682224 21.12 44.100002 4229.0796

Date time could probably be reduced. next is temp of my office, 17.7 would be fine, 21.1 for recroom temp is also fine and next is 44 watts is what my woodstove is using but it always uses 44 watts or 0 so it could be 1 or 0 same with the 4229 watts for the hotwater could be reduced to on or off. Even if I kept the values for applications where they have changing watts rounding to whole numbers is fine.
so

2010/12/30 00:55 17.7 21.1 1 1 which is roughly half the size already. dumping the date out should also be possible with a little thought or at least only entering it once at the start of each day. I think I've seen somewhere where the arduino has a clock in it so I can track time based on it not based on what the computer says the date is.

Once again this is becoming a question of what do I really need it to do. My PC is never shutdown for long periods. 30 minutes to do hardware work or major OS changes now and again maybe but that's about it. I probably don't need to log days worth of data.

[Piwoslaw]

Originally Posted by strider3700

2010/12/30 00:55 17.7 21.1 1 1 which is roughly half the size already. dumping the date out should also be possible with a little thought or at least only entering it once at the start of each day. I think I've seen somewhere where the arduino has a clock in it so I can track time based on it not based on what the computer says the date is.

You could dump the clock. If you are collecting data every 5 minutes, then the first row after a date change is midnight, the next row is 5 past midnight, the 40th row is 40x5 minutes past midnight = 3:20am, etc. And since you'll already be creating a new file for each day, then the date is already in the file's name. That pretty much takes you down to 2 temperatures + 2 bits of info in each line.

EDIT: Actually, those two bits of information can be encoded in the temperature info. If the first temperature is positive (resp. negative), then the first bit is 0 (resp. 1). Same with the second. Of course, this will only work if the temperatures you are measuring will always be above 0, which is true in this case.

Also, if you know how many bits are used for each temperature reading, and what temperatures are possible, you may be able to encode everything as one number, then use a spreadsheet to decode.

[strider3700]

yeah I do horrible things to data to make it fit various needs for a living so there is still a bunch I can do to save space.

On a data note - My wife was sick with what looks like norovirus thursday. Yesterday was hardcore sanitize everything day and the hotwater was flowing. The hot water heater ran for 209 minutes partially due to 4 loads in the washing machine on sanitize. This was 70 minutes more then the previous day. I figured that it was a one time thing but today due to everyone catching up on showers/baths another load of hot laundry and so on we're at 211 minutes with the possibility of one more 3 minute top up before midnight hits. I'm really glad I'm logging and then reviewing the data each day because a few quick measurements here and there just don't come close to telling the whole story.

[strider3700]

It's been 14 days of collecting hotwater usage data. A few things I've noticed.

Bath's use a lot of hotwater. You can usually notice when the kids bath days are.

THe heater runs a fair amount to maintain temperature even when set to a very low 108F. Mine runs for 4 minutes every 3 hours roughly. That works out to 1.52kwh of electricity every day just to maintain water temperatures.
in my case that means an electric on demand heater will pay for itself via standby savings in roughly 13 years... I bet a good insulating blanket will be closer to 1 year. I don't have space to fit one on my current tank so I can't test that to know for sure. If you're running your tank hotter then I am then your payback will be quicker of course.

14 days isn't enough time to get a good average use. I've ranged from 211 minutes ran to 103 minutes ran with 1/3 being over 200, 1/3 being near 150 and 1/3 being near below 110. The average is 147 minutes ran.

The major reason I'm logging this is to get an idea of "normal" use and cost so that I know how much I'm saving when I install the solar hotwater setup and monitor the hotwater tank after that. As it currently stands assuming I get the reasonable 70% heating by solar and the electric rate keeps climbing like it did this year(they've requested the same increase yearly for the next 3 at least) then I'll save $200 this year and $1125 over 5 years which should with any luck have me at break even mid year 5. Savings over 25 years is something near $12,000

[AC_Hacker]

Strider,

I just ordered a couple of current sensors of the type you are using. I also ordered 20 of the 1-wire temp sensors and also 20 lm335 linear temp sensors and a couple of pressure sensors.

I have in mind to data-log my homemade heat-pump so that I can wring maximum efficiency out of it. The current sensor had been a stumbling block for a long time. I tried to hand-wrap a torroidal core, but realized that it wouldn't work so good for AC, what with Power Factor and all.

Great tips.

Thanks for showing the way.

-AC_Hacker