Home energy data logging project

[strider3700]

Yesterday I saw that onterio will be increasing the electric bills 46% over the next 5 years as they phase out coal and go renewable. We don't have coal to phase out but are expecting similar increases in the future. The city recently announced millions in water infrastructure updates and they will be billing that not through property taxes but on water usage. In both cases the more you use the worse the hit will be.

So today I decided it's time to get off my *** and get really serious about consumption logging and reduction. We've done pretty good on the electric front but more can be done I'm sure.

Water wise we discovered a break in the main on the cities side of the meter so they'll be digging that out and fixing it shortly. While spending all this time looking at the meter I decided to throw the reading into my spreadsheet and see how we're doing since I got a bill just a month ago. It reports that we're using a lot of water. 200 gallons a day almost which is comparable to my dead of summer watering 1000 sqft of gardens and 14 fruit trees water usage. So I waited 24 hours, and checked again. Since then we've used 52 gallons. Thats with 2 of us having long showers but no laundry or dishwasher. My best guess is they don't actually read the meter 3 times a year and instead average things for the billing. We'll see.

Anyways since I want to do data logging and realtime monitoring of our usage today I ordered an arduino mega microcontroller kit off of ebay. I'm sure it's a chinese knock off but my morals no longer care so long as it works. the unit was 30% the price of an official one. I also ordered up some hall effect current sensors from a small place in the US. Later tonight maybe I'll pick up a bunch of temp sensors.

The plan is to wire up and monitor the following
Electric dryer consumption
Electric hot water heater consumption
cold room temperature
basement back temperature
basement front temperature
bedroom 1,2,3 upstairs temperature
upstairs living room temperature
outside backyard temperature
water from mains temperature
water from hotwater tank temperature

That is a good start to let me know what is going on beyond what my kill a watt can tell me.

As I've been slowly piecing my solar hotwater setup together(read that as I've got a stack of copper pipe, some glass and a tank) I'll use the arduino as the differential controller and some more logging. In the logging it will track
solar tank top temperature
solar tank bottom temperature
trench temperature
pipe to collectors start temperature
collectors inlet temperature
collector outlet temperature
collector surface temperature
plus more I've forgotten I'm sure.

eventually I'd love to put a water meter with digital outputs in line on the main but they are not cheap last time I looked.

My goal is to use this thread to organize my thoughts and flesh out design as I SLOWLY chunk through all of this. I'm a software guy not electronics so a bunch of this will require some research. Looking at the arduino stuff the code is trivial and I've seen some pretty cool web linking so I'd like to try that. The hardware is not too brutal looking but I'm doing my best to stay in the realm of already been done and here's the step by step tutorial on how to make it work.

The pieces I've selected so far are
hall effect current sensors - Linear BiPolar Hall Effect Current Sensor +/- 80 Amps I ordered 4 shipped for $26.40

maxim ds18b20 one wire temp sensors - 2 free samples so far ebay has them shipped for under $2 each if you order more then 10

the arduino ATmega with a 128x64 LCD screen and a bunch of starter electronics stuff that may be useful since I don't have much of it. Arduino ATmega 1280 128x64 Graphic LCD Mega Starter Kit - eBay (item 150521034176 end time Dec-16-10 04:19:03 PST) $68 after shipping.

When everything arrives I'll do some basic testing and get a couple of temp probes working and measure some electrical currents that I can verify against the kill-a-watt before I hook up the 220 appliances.

I'll have to decide what I want to do wire wise for all of this stuff. for things like the solar panels cat5 may work, but multiple runs of simple twisted pair may be cheaper. We'll see what home depot is carrying.


This should be fun. I've only been wanting to do it for a couple of years now...

[Piwoslaw]

Re water usage, here are a few ideas:

• I've seen analog (dial) type water meters with a wire that sends a signal every time a certain dial turns. They are cheaper than the fancy digital meters, but could be even cheaper if you DIY (similarly to the gas meter in a link you recently shared).
• Install one for your water mains (the utility probably won't you replace their meter, so install your's after it).
• Put another one before your hot water tank, so you'll more or less know how much hot water you're using.
• One more meter just for the toilets. This will tell you how much can be saved by switching to lower flush models and/or reusing grey water to flush.
• And another meter for water that goes outside (yard, trees, car washing, etc.). Now install a rain water system and watch the meter stop moving Metering water for yard usage is becoming popular here, since it allows the sewage bill to be reduced by that amount.

[strider3700]

OK my arduino mega showed up this morning and having not done any electronics since about 1997 I took my time figuring things out. At this moment I have it reading the temperature using a ds18b20 temperature probe.

It turns out that the b is important because I spent 2 hours trying to figure out what was wrong based on following a tutorial for a ds18s20... The CRC code the device reports back is different. So for my and anyone else that wants to know's information here's what I needed to do.

on the ds18b20 pin 1 goes to ground on the arduino. Pin 3 is jumpered to pin 1 so it's also going to ground on the arduino. 5V on the arduino goes to one end of a 4.7kohm resistor and that resistor then goes to pin 2. pin 22 on the arduino goes to pin 2. Thats it wiring wise.

Here is the code I used, hacked directly from a tutorial then hacked up using some more notes I found elsewhere

#include <OneWire.h>

/* DS18b20 Temperature chip i/o */

OneWire  ds(22);  // on pin 22

void setup(void) {
  Serial.begin(9600);
}

void loop(void) {
  int HighByte, LowByte, TReading, SignBit, Tc_100, Whole, Fract;
  byte i;
  byte present = 0;
  byte data[12];
  byte addr[8];
  
  if ( !ds.search(addr)) {
    Serial.print("No more addresses.\n");
    ds.reset_search();
    delay(250);
    return;
  }
  
  Serial.print("R=");
  for( i = 0; i < 8; i++) {
    Serial.print(addr[i], HEX);
    Serial.print(" ");
  }

  if ( OneWire::crc8( addr, 7) != addr[7]) {
      Serial.print("CRC is not valid!\n");
      return;
  }
  
  if ( addr[0] != 0x28) {    //changed the value to 0x28 to recognize ds18b20 
      Serial.print("Device is not a DS18B20 family device.\n");
      return;
  }

  // The DallasTemperature library can do all this work for you!

  ds.reset();
  ds.select(addr);
  ds.write(0x44,1);         // start conversion, with parasite power on at the end
  
  delay(1000);     // maybe 750ms is enough, maybe not
  // we might do a ds.depower() here, but the reset will take care of it.
  
  present = ds.reset();
  ds.select(addr);    
  ds.write(0xBE);         // Read Scratchpad

  Serial.print("P=");
  Serial.print(present,HEX);
  Serial.print(" ");
  for ( i = 0; i < 9; i++) {           // we need 9 bytes
    data[i] = ds.read();
    Serial.print(data[i], HEX);
    Serial.print(" ");
  }
  Serial.print(" CRC=");
  Serial.print( OneWire::crc8( data, 8), HEX);
  Serial.println();
  LowByte = data[0];
  HighByte = data[1];
  TReading = (HighByte << 8) + LowByte;
  SignBit = TReading & 0x8000;  // test most sig bit
  if (SignBit) // negative
  {
    TReading = (TReading ^ 0xffff) + 1; // 2's comp
  }
  Tc_100 = (6 * TReading) + TReading / 4;    // multiply by (100 * 0.0625) or 6.25

  Whole = Tc_100 / 100;  // separate off the whole and fractional portions
  Fract = Tc_100 % 100;


  if (SignBit) // If its negative
  {
     Serial.print("-");
  }
  Serial.print(Whole);
  Serial.print(".");
  if (Fract < 10)
  {
     Serial.print("0");
  }
  Serial.print(Fract);

  Serial.print("\n");
  
}

You will need to install the onewire library but that is easy enough to find with google, then just extract the zip into the correct location which in my case is /sketchbook/libraries

I think my next thing will be getting multiple temp sensors working hopefully that is easy. After that I want to try out those current sensors but that will involve a bit of soldering.

[strider3700]

ok the second temp sensor is trivial. connect pin 1 and pin3 to ground on the arduino, connect pin 2 to pin 22 the same as the other sensor. I didn't even shut the arduino down or update the code. the serial monitor just started reading the second one as soon as it was hooked up.

right now they are reporting that my office is 17.93 or 17.52 C and they are 3 inches apart on the bread board. They are sensitive enough that if I hold my hand a couple inches above them the temperature starts climbing within 5 seconds. Now to order 25 more and track down a crap load of wire to do the entire house. This is too cool.

Next step I think will be getting this data logging and accessible online. If I track down some 3 wire or an extra cat 5 cable I'll try the current sensors out but that will take some non-coding work.

<edit> Ok I just ran across this page http://www.pachube.com/ it looks like it's tailor made for what I want to do here. I'll have to poke around and see if It will do what I want or if I need to roll my own. </edit>

<edit #2 > apparently pachube is not free if you want to store data for more then 30 days. the non free options range from really expensive to insanely expensive. $15/month to $900/month with 100% storage starting at $100/month... I'll play with it a bit for now but google apps is likely in my near future </edit #2>

[strider3700]

Ok getting data logging going on even at the simplest level turned out to be stupidly difficult for me. I've only been programming for a couple of decades now but I don't think I've ever touched the serial port so I decided to just grab an example someone else did and go with that.

My initial goal is just to get it writing every output to a local file. Nothing hard at all right?

Well I first tried a simple batch script that just monitored the serial port but that got a lot of corrupted crap or dropped data. So I figured since the arduino is using python lets just find a python script and do that. Well 4 hours and 5 examples later it wasn't working correctly. Once again lots of errors and corrupted data. I'm also not a python guy so debugging not knowing the language was a nightmare.

So I went to java which isn't my day to day but I've done a bunch of and it's damn close to c# which I mostly do these days. Then I figured since I have a java example lets learn eclipse, I've heard good things about it. Well it's pretty and shiny but it took me awhile to figure out how to add a jar to the project to make the rxtx library work. Anyways here's the code I found after I changed the port to be correct

package datalogger;

	import java.io.InputStream;
	import java.io.OutputStream;
	import gnu.io.CommPortIdentifier; 
	import gnu.io.SerialPort;
	import gnu.io.SerialPortEvent; 
	import gnu.io.SerialPortEventListener; 
	import java.util.Enumeration;

	public class SerialTest implements SerialPortEventListener {
		SerialPort serialPort;
	        /** The port we're normally going to use. */
		private static final String PORT_NAMES[] = { 
								"/dev/ttyUSB0", // Linux
								};
		/** Buffered input stream from the port */
		private InputStream input;
		/** The output stream to the port */
		private OutputStream output;
		/** Milliseconds to block while waiting for port open */
		private static final int TIME_OUT = 2000;
		/** Default bits per second for COM port. */
		private static final int DATA_RATE = 9600;

		public void initialize() {
			CommPortIdentifier portId = null;
			Enumeration portEnum = CommPortIdentifier.getPortIdentifiers();

			// iterate through, looking for the port
			while (portEnum.hasMoreElements()) {
				CommPortIdentifier currPortId = (CommPortIdentifier) portEnum.nextElement();
				for (String portName : PORT_NAMES) {
					if (currPortId.getName().equals(portName)) {
						portId = currPortId;
						break;
					}
				}
			}

			if (portId == null) {
				System.out.println("Could not find COM port.");
				return;
			}

			try {
				// open serial port, and use class name for the appName.
				serialPort = (SerialPort) portId.open(this.getClass().getName(),
						TIME_OUT);

				// set port parameters
				serialPort.setSerialPortParams(DATA_RATE,
						SerialPort.DATABITS_8,
						SerialPort.STOPBITS_1,
						SerialPort.PARITY_NONE);

				// open the streams
				input = serialPort.getInputStream();
				output = serialPort.getOutputStream();

				// add event listeners
				serialPort.addEventListener(this);
				serialPort.notifyOnDataAvailable(true);
			} catch (Exception e) {
				System.err.println(e.toString());
			}
		}

		/**
		 * This should be called when you stop using the port.
		 * This will prevent port locking on platforms like Linux.
		 */
		public synchronized void close() {
			if (serialPort != null) {
				serialPort.removeEventListener();
				serialPort.close();
			}
		}

		/**
		 * Handle an event on the serial port. Read the data and print it.
		 */
		public synchronized void serialEvent(SerialPortEvent oEvent) {
			if (oEvent.getEventType() == SerialPortEvent.DATA_AVAILABLE) {
				try {
					int available = input.available();
					byte chunk[] = new byte[available];
					input.read(chunk, 0, available);

					// Displayed results are codepage dependent
					System.out.print(new String(chunk));
				} catch (Exception e) {
					System.err.println(e.toString());
				}
			}
			// Ignore all the other eventTypes, but you should consider the other ones.
		}
	public static void main(String[] args) {
		SerialTest main = new SerialTest();
		main.initialize();
		System.out.println("Started");
	}

}

I got it from here. Arduino playground - Java
I like that it's event driven it totally gets rid of all those issues I had with the other attempts. Anyways the hard part was adding the rxtx jar recognized in eclipse. I followed this partially (it's for windows I'm on linux) Using RXTX In Eclipse - Rxtx. That got it working so I can now go back to some old school reading and writing to disk...

I think the major Issue I've had with getting this part working is there are hundreds of ways to do it and none of them are amazingly well documented. The arduino people of course don't recommend one method over another and google will net you thousands of partial chunks of code pointing you in the right direction but none of them are plug and play.

[Daox]

While I haven't taken the time to really dig into your posts on this yet, please keep posting as it does look quite interesting!

[strider3700]

I have to say I love this thing so far. Last night it logged temperatures in my office every minute to a csv file. This morning I threw it into a spreadsheet and charted it.



The spikes every 15 minutes or so are the circulating fan for the furnace kicking on. The heat pump is not actively heating the fan is just helping to move the hot air from the woodstove around. I was never sure if it was actually effective but it is obviously pushing a bit of hot air in here.

THat last spike is also something I've noticed manually watching the sensor data. Thats me coming into the office and sitting down 18 inches from the sensor without a shirt on. People release a lot of heat.

Tonights goal will be to get the java program parsing the incoming stream better. Right now I just raw right it to a file and that is handling the new lines and putting the chunks together. I want to get it actually parsing first of all complete lines then split those lines up so that I can start doing averaging and so on inside of the program.

Also Sensor 282895C50200009B is a really crappy label. That is one of my complaints with the one wire system so far. I will need to install each sensor record it's ID in HEX then label it so that when I have 20 of them out there I know which room has which sensor so that I can apply a label to them. If not it's going to be a game of put your hand on it and sit here watching for the temp to go up then apply the label.

[strider3700]

To not screw up my simple logging and get an easy pretty graph I had turned off the second sensor reading on the arduino rather then parse it on the java program. Last night I turned it back on and discovered I had an issue. The Event that fires saying data is on the serial port grab it would happen quicker then the java program could parse the first event so basically I'd get the entire first sensor in except for the new line character and then the second sensor would partially send garbling the results. I went to bed thinking that this would suck to handle, I'd need a second thread to process data and one to just grab the data. This morning I woke up realizing it's far far easier to just add a delay between sending each sensors data. so the send part on my arduino now looks like this.

printData(s1);
delay(100);
printData(s2);

It's a hack but it works. I'll play around with the delay to see how short I can get it without causing issues. I don't want to take 3 seconds to read in 30 sensors worth of data if possible....

<edit> A delay of 30 between each sensor write is good enough to allow me to output 100 sensors worth of readings. I've moved the delay into the print data function on the arduino. </edit>

[strider3700]

And last nights office temps

You can clearly see the effects of me closing the office door right at the start and then opening it in the morning right at the end. THe gradual middle of the night climb is from me getting the fire going better.

[strider3700]

My last edit reminded me. I started using the dallas temperatures library on the arduino so it's code is very different and much cleaner. It's still in bodge it together to see if it works stage though so it's far from pretty but here is the complete arduino code as it currently stands.

#include <OneWire.h>
#include <DallasTemperature.h>

// Data wire is plugged into port 22 on the Arduino
#define ONE_WIRE_BUS 22
#define TEMPERATURE_PRECISION 9
#define delayTime 2000     // the delay before restarting the loop

// Setup a oneWire instance to communicate with any OneWire devices (not just Maxim/Dallas temperature ICs)
OneWire oneWire(ONE_WIRE_BUS);

// Pass our oneWire reference to Dallas Temperature. 
DallasTemperature sensors(&oneWire);

// arrays to hold device addresses
DeviceAddress s1, s2;

void setup(void)
{
  // start serial port
  Serial.begin(9600);
  
  // Start up the library
  sensors.begin();

  //printStatus();

  // assign address manually.  the addresses below will beed to be changed
  // to valid device addresses on your bus.  device address can be retrieved
  // by using either oneWire.search(deviceAddress) or individually via
  // sensors.getAddress(deviceAddress, index)
  //insideThermometer = { 0x28, 0x1D, 0x39, 0x31, 0x2, 0x0, 0x0, 0xF0 };
  //outsideThermometer   = { 0x28, 0x3F, 0x1C, 0x31, 0x2, 0x0, 0x0, 0x2 };

  // search for devices on the bus and assign based on an index.  ideally,
  // you would do this to initially discover addresses on the bus and then 
  // use those addresses and manually assign them (see above) once you know 
  // the devices on your bus (and assuming they don't change).
  // 
  // method 1: by index
  if (!sensors.getAddress(s1, 0)) Serial.println("Unable to find address for Device 0"); 
  if (!sensors.getAddress(s2, 1)) Serial.println("Unable to find address for Device 1"); 

  // method 2: search()
  // search() looks for the next device. Returns 1 if a new address has been
  // returned. A zero might mean that the bus is shorted, there are no devices, 
  // or you have already retrieved all of them.  It might be a good idea to 
  // check the CRC to make sure you didn't get garbage.  The order is 
  // deterministic. You will always get the same devices in the same order
  //
  // Must be called before search()
  //oneWire.reset_search();
  // assigns the first address found to insideThermometer
  //if (!oneWire.search(insideThermometer)) Serial.println("Unable to find address for insideThermometer");
  // assigns the seconds address found to outsideThermometer
  //if (!oneWire.search(outsideThermometer)) Serial.println("Unable to find address for outsideThermometer");

  
  // set the resolution to 12 bit
  sensors.setResolution(s1, 12);
  sensors.setResolution(s2, 12);

  
}

// function to print a device address
void printAddress(DeviceAddress deviceAddress)
{
  for (uint8_t i = 0; i < 8; i++)
  {
    // zero pad the address if necessary
    if (deviceAddress[i] < 16) Serial.print("0");
    Serial.print(deviceAddress[i], HEX);
  }
}

void printStatus(void)
{
  // locate devices on the bus
  Serial.print("Locating devices...");
  Serial.print("Found ");
  Serial.print(sensors.getDeviceCount(), DEC);
  Serial.println(" devices.");

  // report parasite power requirements
  Serial.print("Parasite power is: "); 
  if (sensors.isParasitePowerMode()) Serial.println("ON");
  else Serial.println("OFF");
  
  // show the addresses we found on the bus
  Serial.print("Device 0 Address: ");
  printAddress(s1);
  Serial.println();

  Serial.print("Device 1 Address: ");
  printAddress(s2);
  Serial.println();
  
  // print the device resolutions
  Serial.print("Device 0 Resolution: ");
  Serial.print(sensors.getResolution(s1), DEC); 
  Serial.println();

  Serial.print("Device 1 Resolution: ");
  Serial.print(sensors.getResolution(s2), DEC); 
  Serial.println();
}
// function to print the temperature for a device
void printTemperature(DeviceAddress deviceAddress)
{
  float tempC = sensors.getTempC(deviceAddress);
  //Serial.print("Temp C: ");
  Serial.print(tempC);
}

// function to print a device's resolution
void printResolution(DeviceAddress deviceAddress)
{
  Serial.print("Resolution: ");
  Serial.print(sensors.getResolution(deviceAddress));
  Serial.println();    
}

// main function to print information about a device
void printData(DeviceAddress deviceAddress)
{
  //Serial.print("Device Address: ");
  printAddress(deviceAddress);
  Serial.print(",");
  printTemperature(deviceAddress);
  Serial.println();
  delay(30);
}

void loop(void)
{ 
  // call sensors.requestTemperatures() to issue a global temperature 
  // request to all devices on the bus
  //Serial.print("Requesting temperatures...");
  sensors.requestTemperatures();
  //Serial.println("DONE");

  // print the device information
  printData(s1);
  printData(s2);
  delay(delayTime);
}