11-19-10, 06:10 PM | #1 |
Master EcoRenovator
Join Date: Dec 2008
Location: Vancouver Island BC
Posts: 745
Thanks: 23
Thanked 37 Times in 30 Posts
|
Home energy data logging project
Yesterday I saw that onterio will be increasing the electric bills 46% over the next 5 years as they phase out coal and go renewable. We don't have coal to phase out but are expecting similar increases in the future. The city recently announced millions in water infrastructure updates and they will be billing that not through property taxes but on water usage. In both cases the more you use the worse the hit will be.
So today I decided it's time to get off my *** and get really serious about consumption logging and reduction. We've done pretty good on the electric front but more can be done I'm sure. Water wise we discovered a break in the main on the cities side of the meter so they'll be digging that out and fixing it shortly. While spending all this time looking at the meter I decided to throw the reading into my spreadsheet and see how we're doing since I got a bill just a month ago. It reports that we're using a lot of water. 200 gallons a day almost which is comparable to my dead of summer watering 1000 sqft of gardens and 14 fruit trees water usage. So I waited 24 hours, and checked again. Since then we've used 52 gallons. Thats with 2 of us having long showers but no laundry or dishwasher. My best guess is they don't actually read the meter 3 times a year and instead average things for the billing. We'll see. Anyways since I want to do data logging and realtime monitoring of our usage today I ordered an arduino mega microcontroller kit off of ebay. I'm sure it's a chinese knock off but my morals no longer care so long as it works. the unit was 30% the price of an official one. I also ordered up some hall effect current sensors from a small place in the US. Later tonight maybe I'll pick up a bunch of temp sensors. The plan is to wire up and monitor the following Electric dryer consumption Electric hot water heater consumption cold room temperature basement back temperature basement front temperature bedroom 1,2,3 upstairs temperature upstairs living room temperature outside backyard temperature water from mains temperature water from hotwater tank temperature That is a good start to let me know what is going on beyond what my kill a watt can tell me. As I've been slowly piecing my solar hotwater setup together(read that as I've got a stack of copper pipe, some glass and a tank) I'll use the arduino as the differential controller and some more logging. In the logging it will track solar tank top temperature solar tank bottom temperature trench temperature pipe to collectors start temperature collectors inlet temperature collector outlet temperature collector surface temperature plus more I've forgotten I'm sure. eventually I'd love to put a water meter with digital outputs in line on the main but they are not cheap last time I looked. My goal is to use this thread to organize my thoughts and flesh out design as I SLOWLY chunk through all of this. I'm a software guy not electronics so a bunch of this will require some research. Looking at the arduino stuff the code is trivial and I've seen some pretty cool web linking so I'd like to try that. The hardware is not too brutal looking but I'm doing my best to stay in the realm of already been done and here's the step by step tutorial on how to make it work. The pieces I've selected so far are hall effect current sensors - Linear BiPolar Hall Effect Current Sensor +/- 80 Amps I ordered 4 shipped for $26.40 maxim ds18b20 one wire temp sensors - 2 free samples so far ebay has them shipped for under $2 each if you order more then 10 the arduino ATmega with a 128x64 LCD screen and a bunch of starter electronics stuff that may be useful since I don't have much of it. Arduino ATmega 1280 128x64 Graphic LCD Mega Starter Kit - eBay (item 150521034176 end time Dec-16-10 04:19:03 PST) $68 after shipping. When everything arrives I'll do some basic testing and get a couple of temp probes working and measure some electrical currents that I can verify against the kill-a-watt before I hook up the 220 appliances. I'll have to decide what I want to do wire wise for all of this stuff. for things like the solar panels cat5 may work, but multiple runs of simple twisted pair may be cheaper. We'll see what home depot is carrying. This should be fun. I've only been wanting to do it for a couple of years now... |
11-20-10, 01:02 AM | #2 |
Super Moderator
Join Date: May 2009
Location: Warsaw, Poland
Posts: 964
Thanks: 189
Thanked 111 Times in 87 Posts
|
Re water usage, here are a few ideas:
|
12-07-10, 03:35 AM | #3 |
Master EcoRenovator
Join Date: Dec 2008
Location: Vancouver Island BC
Posts: 745
Thanks: 23
Thanked 37 Times in 30 Posts
|
OK my arduino mega showed up this morning and having not done any electronics since about 1997 I took my time figuring things out. At this moment I have it reading the temperature using a ds18b20 temperature probe.
It turns out that the b is important because I spent 2 hours trying to figure out what was wrong based on following a tutorial for a ds18s20... The CRC code the device reports back is different. So for my and anyone else that wants to know's information here's what I needed to do. on the ds18b20 pin 1 goes to ground on the arduino. Pin 3 is jumpered to pin 1 so it's also going to ground on the arduino. 5V on the arduino goes to one end of a 4.7kohm resistor and that resistor then goes to pin 2. pin 22 on the arduino goes to pin 2. Thats it wiring wise. Here is the code I used, hacked directly from a tutorial then hacked up using some more notes I found elsewhere Code:
#include <OneWire.h> /* DS18b20 Temperature chip i/o */ OneWire ds(22); // on pin 22 void setup(void) { Serial.begin(9600); } void loop(void) { int HighByte, LowByte, TReading, SignBit, Tc_100, Whole, Fract; byte i; byte present = 0; byte data[12]; byte addr[8]; if ( !ds.search(addr)) { Serial.print("No more addresses.\n"); ds.reset_search(); delay(250); return; } Serial.print("R="); for( i = 0; i < 8; i++) { Serial.print(addr[i], HEX); Serial.print(" "); } if ( OneWire::crc8( addr, 7) != addr[7]) { Serial.print("CRC is not valid!\n"); return; } if ( addr[0] != 0x28) { //changed the value to 0x28 to recognize ds18b20 Serial.print("Device is not a DS18B20 family device.\n"); return; } // The DallasTemperature library can do all this work for you! ds.reset(); ds.select(addr); ds.write(0x44,1); // start conversion, with parasite power on at the end delay(1000); // maybe 750ms is enough, maybe not // we might do a ds.depower() here, but the reset will take care of it. present = ds.reset(); ds.select(addr); ds.write(0xBE); // Read Scratchpad Serial.print("P="); Serial.print(present,HEX); Serial.print(" "); for ( i = 0; i < 9; i++) { // we need 9 bytes data[i] = ds.read(); Serial.print(data[i], HEX); Serial.print(" "); } Serial.print(" CRC="); Serial.print( OneWire::crc8( data, 8), HEX); Serial.println(); LowByte = data[0]; HighByte = data[1]; TReading = (HighByte << 8) + LowByte; SignBit = TReading & 0x8000; // test most sig bit if (SignBit) // negative { TReading = (TReading ^ 0xffff) + 1; // 2's comp } Tc_100 = (6 * TReading) + TReading / 4; // multiply by (100 * 0.0625) or 6.25 Whole = Tc_100 / 100; // separate off the whole and fractional portions Fract = Tc_100 % 100; if (SignBit) // If its negative { Serial.print("-"); } Serial.print(Whole); Serial.print("."); if (Fract < 10) { Serial.print("0"); } Serial.print(Fract); Serial.print("\n"); } I think my next thing will be getting multiple temp sensors working hopefully that is easy. After that I want to try out those current sensors but that will involve a bit of soldering. |
12-07-10, 02:53 PM | #4 |
Master EcoRenovator
Join Date: Dec 2008
Location: Vancouver Island BC
Posts: 745
Thanks: 23
Thanked 37 Times in 30 Posts
|
ok the second temp sensor is trivial. connect pin 1 and pin3 to ground on the arduino, connect pin 2 to pin 22 the same as the other sensor. I didn't even shut the arduino down or update the code. the serial monitor just started reading the second one as soon as it was hooked up.
right now they are reporting that my office is 17.93 or 17.52 C and they are 3 inches apart on the bread board. They are sensitive enough that if I hold my hand a couple inches above them the temperature starts climbing within 5 seconds. Now to order 25 more and track down a crap load of wire to do the entire house. This is too cool. Next step I think will be getting this data logging and accessible online. If I track down some 3 wire or an extra cat 5 cable I'll try the current sensors out but that will take some non-coding work. <edit> Ok I just ran across this page http://www.pachube.com/ it looks like it's tailor made for what I want to do here. I'll have to poke around and see if It will do what I want or if I need to roll my own. </edit> <edit #2 > apparently pachube is not free if you want to store data for more then 30 days. the non free options range from really expensive to insanely expensive. $15/month to $900/month with 100% storage starting at $100/month... I'll play with it a bit for now but google apps is likely in my near future </edit #2> Last edited by strider3700; 12-07-10 at 03:05 PM.. Reason: found pachube |
12-09-10, 12:27 AM | #5 |
Master EcoRenovator
Join Date: Dec 2008
Location: Vancouver Island BC
Posts: 745
Thanks: 23
Thanked 37 Times in 30 Posts
|
Ok getting data logging going on even at the simplest level turned out to be stupidly difficult for me. I've only been programming for a couple of decades now but I don't think I've ever touched the serial port so I decided to just grab an example someone else did and go with that.
My initial goal is just to get it writing every output to a local file. Nothing hard at all right? Well I first tried a simple batch script that just monitored the serial port but that got a lot of corrupted crap or dropped data. So I figured since the arduino is using python lets just find a python script and do that. Well 4 hours and 5 examples later it wasn't working correctly. Once again lots of errors and corrupted data. I'm also not a python guy so debugging not knowing the language was a nightmare. So I went to java which isn't my day to day but I've done a bunch of and it's damn close to c# which I mostly do these days. Then I figured since I have a java example lets learn eclipse, I've heard good things about it. Well it's pretty and shiny but it took me awhile to figure out how to add a jar to the project to make the rxtx library work. Anyways here's the code I found after I changed the port to be correct Code:
package datalogger; import java.io.InputStream; import java.io.OutputStream; import gnu.io.CommPortIdentifier; import gnu.io.SerialPort; import gnu.io.SerialPortEvent; import gnu.io.SerialPortEventListener; import java.util.Enumeration; public class SerialTest implements SerialPortEventListener { SerialPort serialPort; /** The port we're normally going to use. */ private static final String PORT_NAMES[] = { "/dev/ttyUSB0", // Linux }; /** Buffered input stream from the port */ private InputStream input; /** The output stream to the port */ private OutputStream output; /** Milliseconds to block while waiting for port open */ private static final int TIME_OUT = 2000; /** Default bits per second for COM port. */ private static final int DATA_RATE = 9600; public void initialize() { CommPortIdentifier portId = null; Enumeration portEnum = CommPortIdentifier.getPortIdentifiers(); // iterate through, looking for the port while (portEnum.hasMoreElements()) { CommPortIdentifier currPortId = (CommPortIdentifier) portEnum.nextElement(); for (String portName : PORT_NAMES) { if (currPortId.getName().equals(portName)) { portId = currPortId; break; } } } if (portId == null) { System.out.println("Could not find COM port."); return; } try { // open serial port, and use class name for the appName. serialPort = (SerialPort) portId.open(this.getClass().getName(), TIME_OUT); // set port parameters serialPort.setSerialPortParams(DATA_RATE, SerialPort.DATABITS_8, SerialPort.STOPBITS_1, SerialPort.PARITY_NONE); // open the streams input = serialPort.getInputStream(); output = serialPort.getOutputStream(); // add event listeners serialPort.addEventListener(this); serialPort.notifyOnDataAvailable(true); } catch (Exception e) { System.err.println(e.toString()); } } /** * This should be called when you stop using the port. * This will prevent port locking on platforms like Linux. */ public synchronized void close() { if (serialPort != null) { serialPort.removeEventListener(); serialPort.close(); } } /** * Handle an event on the serial port. Read the data and print it. */ public synchronized void serialEvent(SerialPortEvent oEvent) { if (oEvent.getEventType() == SerialPortEvent.DATA_AVAILABLE) { try { int available = input.available(); byte chunk[] = new byte[available]; input.read(chunk, 0, available); // Displayed results are codepage dependent System.out.print(new String(chunk)); } catch (Exception e) { System.err.println(e.toString()); } } // Ignore all the other eventTypes, but you should consider the other ones. } public static void main(String[] args) { SerialTest main = new SerialTest(); main.initialize(); System.out.println("Started"); } } I like that it's event driven it totally gets rid of all those issues I had with the other attempts. Anyways the hard part was adding the rxtx jar recognized in eclipse. I followed this partially (it's for windows I'm on linux) Using RXTX In Eclipse - Rxtx. That got it working so I can now go back to some old school reading and writing to disk... I think the major Issue I've had with getting this part working is there are hundreds of ways to do it and none of them are amazingly well documented. The arduino people of course don't recommend one method over another and google will net you thousands of partial chunks of code pointing you in the right direction but none of them are plug and play. |
12-09-10, 06:08 AM | #6 |
Administrator
Join Date: Aug 2008
Location: Germantown, WI
Posts: 5,525
Thanks: 1,162
Thanked 374 Times in 305 Posts
|
While I haven't taken the time to really dig into your posts on this yet, please keep posting as it does look quite interesting!
__________________
Current project - To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. & To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. |
12-09-10, 12:06 PM | #7 |
Master EcoRenovator
Join Date: Dec 2008
Location: Vancouver Island BC
Posts: 745
Thanks: 23
Thanked 37 Times in 30 Posts
|
I have to say I love this thing so far. Last night it logged temperatures in my office every minute to a csv file. This morning I threw it into a spreadsheet and charted it.
The spikes every 15 minutes or so are the circulating fan for the furnace kicking on. The heat pump is not actively heating the fan is just helping to move the hot air from the woodstove around. I was never sure if it was actually effective but it is obviously pushing a bit of hot air in here. THat last spike is also something I've noticed manually watching the sensor data. Thats me coming into the office and sitting down 18 inches from the sensor without a shirt on. People release a lot of heat. Tonights goal will be to get the java program parsing the incoming stream better. Right now I just raw right it to a file and that is handling the new lines and putting the chunks together. I want to get it actually parsing first of all complete lines then split those lines up so that I can start doing averaging and so on inside of the program. Also Sensor 282895C50200009B is a really crappy label. That is one of my complaints with the one wire system so far. I will need to install each sensor record it's ID in HEX then label it so that when I have 20 of them out there I know which room has which sensor so that I can apply a label to them. If not it's going to be a game of put your hand on it and sit here watching for the temp to go up then apply the label. |
12-10-10, 01:17 PM | #8 |
Master EcoRenovator
Join Date: Dec 2008
Location: Vancouver Island BC
Posts: 745
Thanks: 23
Thanked 37 Times in 30 Posts
|
To not screw up my simple logging and get an easy pretty graph I had turned off the second sensor reading on the arduino rather then parse it on the java program. Last night I turned it back on and discovered I had an issue. The Event that fires saying data is on the serial port grab it would happen quicker then the java program could parse the first event so basically I'd get the entire first sensor in except for the new line character and then the second sensor would partially send garbling the results. I went to bed thinking that this would suck to handle, I'd need a second thread to process data and one to just grab the data. This morning I woke up realizing it's far far easier to just add a delay between sending each sensors data. so the send part on my arduino now looks like this.
Code:
printData(s1); delay(100); printData(s2); <edit> A delay of 30 between each sensor write is good enough to allow me to output 100 sensors worth of readings. I've moved the delay into the print data function on the arduino. </edit> Last edited by strider3700; 12-10-10 at 01:45 PM.. Reason: figured out an acceptable delay |
12-10-10, 01:32 PM | #9 |
Master EcoRenovator
Join Date: Dec 2008
Location: Vancouver Island BC
Posts: 745
Thanks: 23
Thanked 37 Times in 30 Posts
|
And last nights office temps
You can clearly see the effects of me closing the office door right at the start and then opening it in the morning right at the end. THe gradual middle of the night climb is from me getting the fire going better. |
12-10-10, 01:47 PM | #10 |
Master EcoRenovator
Join Date: Dec 2008
Location: Vancouver Island BC
Posts: 745
Thanks: 23
Thanked 37 Times in 30 Posts
|
My last edit reminded me. I started using the dallas temperatures library on the arduino so it's code is very different and much cleaner. It's still in bodge it together to see if it works stage though so it's far from pretty but here is the complete arduino code as it currently stands.
Code:
#include <OneWire.h> #include <DallasTemperature.h> // Data wire is plugged into port 22 on the Arduino #define ONE_WIRE_BUS 22 #define TEMPERATURE_PRECISION 9 #define delayTime 2000 // the delay before restarting the loop // Setup a oneWire instance to communicate with any OneWire devices (not just Maxim/Dallas temperature ICs) OneWire oneWire(ONE_WIRE_BUS); // Pass our oneWire reference to Dallas Temperature. DallasTemperature sensors(&oneWire); // arrays to hold device addresses DeviceAddress s1, s2; void setup(void) { // start serial port Serial.begin(9600); // Start up the library sensors.begin(); //printStatus(); // assign address manually. the addresses below will beed to be changed // to valid device addresses on your bus. device address can be retrieved // by using either oneWire.search(deviceAddress) or individually via // sensors.getAddress(deviceAddress, index) //insideThermometer = { 0x28, 0x1D, 0x39, 0x31, 0x2, 0x0, 0x0, 0xF0 }; //outsideThermometer = { 0x28, 0x3F, 0x1C, 0x31, 0x2, 0x0, 0x0, 0x2 }; // search for devices on the bus and assign based on an index. ideally, // you would do this to initially discover addresses on the bus and then // use those addresses and manually assign them (see above) once you know // the devices on your bus (and assuming they don't change). // // method 1: by index if (!sensors.getAddress(s1, 0)) Serial.println("Unable to find address for Device 0"); if (!sensors.getAddress(s2, 1)) Serial.println("Unable to find address for Device 1"); // method 2: search() // search() looks for the next device. Returns 1 if a new address has been // returned. A zero might mean that the bus is shorted, there are no devices, // or you have already retrieved all of them. It might be a good idea to // check the CRC to make sure you didn't get garbage. The order is // deterministic. You will always get the same devices in the same order // // Must be called before search() //oneWire.reset_search(); // assigns the first address found to insideThermometer //if (!oneWire.search(insideThermometer)) Serial.println("Unable to find address for insideThermometer"); // assigns the seconds address found to outsideThermometer //if (!oneWire.search(outsideThermometer)) Serial.println("Unable to find address for outsideThermometer"); // set the resolution to 12 bit sensors.setResolution(s1, 12); sensors.setResolution(s2, 12); } // function to print a device address void printAddress(DeviceAddress deviceAddress) { for (uint8_t i = 0; i < 8; i++) { // zero pad the address if necessary if (deviceAddress[i] < 16) Serial.print("0"); Serial.print(deviceAddress[i], HEX); } } void printStatus(void) { // locate devices on the bus Serial.print("Locating devices..."); Serial.print("Found "); Serial.print(sensors.getDeviceCount(), DEC); Serial.println(" devices."); // report parasite power requirements Serial.print("Parasite power is: "); if (sensors.isParasitePowerMode()) Serial.println("ON"); else Serial.println("OFF"); // show the addresses we found on the bus Serial.print("Device 0 Address: "); printAddress(s1); Serial.println(); Serial.print("Device 1 Address: "); printAddress(s2); Serial.println(); // print the device resolutions Serial.print("Device 0 Resolution: "); Serial.print(sensors.getResolution(s1), DEC); Serial.println(); Serial.print("Device 1 Resolution: "); Serial.print(sensors.getResolution(s2), DEC); Serial.println(); } // function to print the temperature for a device void printTemperature(DeviceAddress deviceAddress) { float tempC = sensors.getTempC(deviceAddress); //Serial.print("Temp C: "); Serial.print(tempC); } // function to print a device's resolution void printResolution(DeviceAddress deviceAddress) { Serial.print("Resolution: "); Serial.print(sensors.getResolution(deviceAddress)); Serial.println(); } // main function to print information about a device void printData(DeviceAddress deviceAddress) { //Serial.print("Device Address: "); printAddress(deviceAddress); Serial.print(","); printTemperature(deviceAddress); Serial.println(); delay(30); } void loop(void) { // call sensors.requestTemperatures() to issue a global temperature // request to all devices on the bus //Serial.print("Requesting temperatures..."); sensors.requestTemperatures(); //Serial.println("DONE"); // print the device information printData(s1); printData(s2); delay(delayTime); } |
Tags |
data, logging |
Thread Tools | |
Display Modes | |
|
|