EcoRenovator
Page 1 of 2 1 2
Show 40 post(s) from this thread on one page

EcoRenovator (https://ecorenovator.org/forum/index.php)
-   Appliances & Gadgets (https://ecorenovator.org/forum/forumdisplay.php?f=21)
-   -   Home energy data logging project (https://ecorenovator.org/forum/showthread.php?t=1276)

strider3700 11-19-10 06:10 PM
Home energy data logging project
 
Yesterday I saw that onterio will be increasing the electric bills 46% over the next 5 years as they phase out coal and go renewable. We don't have coal to phase out but are expecting similar increases in the future. The city recently announced millions in water infrastructure updates and they will be billing that not through property taxes but on water usage. In both cases the more you use the worse the hit will be.

So today I decided it's time to get off my *** and get really serious about consumption logging and reduction. We've done pretty good on the electric front but more can be done I'm sure.

Water wise we discovered a break in the main on the cities side of the meter so they'll be digging that out and fixing it shortly. While spending all this time looking at the meter I decided to throw the reading into my spreadsheet and see how we're doing since I got a bill just a month ago. It reports that we're using a lot of water. 200 gallons a day almost which is comparable to my dead of summer watering 1000 sqft of gardens and 14 fruit trees water usage. So I waited 24 hours, and checked again. Since then we've used 52 gallons. Thats with 2 of us having long showers but no laundry or dishwasher. My best guess is they don't actually read the meter 3 times a year and instead average things for the billing. We'll see.

Anyways since I want to do data logging and realtime monitoring of our usage today I ordered an arduino mega microcontroller kit off of ebay. I'm sure it's a chinese knock off but my morals no longer care so long as it works. the unit was 30% the price of an official one. I also ordered up some hall effect current sensors from a small place in the US. Later tonight maybe I'll pick up a bunch of temp sensors.

The plan is to wire up and monitor the following
Electric dryer consumption
Electric hot water heater consumption
cold room temperature
basement back temperature
basement front temperature
bedroom 1,2,3 upstairs temperature
upstairs living room temperature
outside backyard temperature
water from mains temperature
water from hotwater tank temperature

That is a good start to let me know what is going on beyond what my kill a watt can tell me.

As I've been slowly piecing my solar hotwater setup together(read that as I've got a stack of copper pipe, some glass and a tank) I'll use the arduino as the differential controller and some more logging. In the logging it will track
solar tank top temperature
solar tank bottom temperature
trench temperature
pipe to collectors start temperature
collectors inlet temperature
collector outlet temperature
collector surface temperature
plus more I've forgotten I'm sure.

eventually I'd love to put a water meter with digital outputs in line on the main but they are not cheap last time I looked.

My goal is to use this thread to organize my thoughts and flesh out design as I SLOWLY chunk through all of this. I'm a software guy not electronics so a bunch of this will require some research. Looking at the arduino stuff the code is trivial and I've seen some pretty cool web linking so I'd like to try that. The hardware is not too brutal looking but I'm doing my best to stay in the realm of already been done and here's the step by step tutorial on how to make it work.

The pieces I've selected so far are
hall effect current sensors - Linear BiPolar Hall Effect Current Sensor +/- 80 Amps I ordered 4 shipped for $26.40

maxim ds18b20 one wire temp sensors - 2 free samples so far ebay has them shipped for under $2 each if you order more then 10

the arduino ATmega with a 128x64 LCD screen and a bunch of starter electronics stuff that may be useful since I don't have much of it. Arduino ATmega 1280 128x64 Graphic LCD Mega Starter Kit - eBay (item 150521034176 end time Dec-16-10 04:19:03 PST) $68 after shipping.

When everything arrives I'll do some basic testing and get a couple of temp probes working and measure some electrical currents that I can verify against the kill-a-watt before I hook up the 220 appliances.

I'll have to decide what I want to do wire wise for all of this stuff. for things like the solar panels cat5 may work, but multiple runs of simple twisted pair may be cheaper. We'll see what home depot is carrying.


This should be fun. I've only been wanting to do it for a couple of years now...

Piwoslaw 11-20-10 01:02 AM
Re water usage, here are a few ideas:
  • I've seen analog (dial) type water meters with a wire that sends a signal every time a certain dial turns. They are cheaper than the fancy digital meters, but could be even cheaper if you DIY (similarly to the gas meter in a link you recently shared).
  • Install one for your water mains (the utility probably won't you replace their meter, so install your's after it).
  • Put another one before your hot water tank, so you'll more or less know how much hot water you're using.
  • One more meter just for the toilets. This will tell you how much can be saved by switching to lower flush models and/or reusing grey water to flush.
  • And another meter for water that goes outside (yard, trees, car washing, etc.). Now install a rain water system and watch the meter stop moving:D Metering water for yard usage is becoming popular here, since it allows the sewage bill to be reduced by that amount.

strider3700 12-07-10 03:35 AM
OK my arduino mega showed up this morning and having not done any electronics since about 1997 I took my time figuring things out. At this moment I have it reading the temperature using a ds18b20 temperature probe.

It turns out that the b is important because I spent 2 hours trying to figure out what was wrong based on following a tutorial for a ds18s20... The CRC code the device reports back is different. So for my and anyone else that wants to know's information here's what I needed to do.

on the ds18b20 pin 1 goes to ground on the arduino. Pin 3 is jumpered to pin 1 so it's also going to ground on the arduino. 5V on the arduino goes to one end of a 4.7kohm resistor and that resistor then goes to pin 2. pin 22 on the arduino goes to pin 2. Thats it wiring wise.

Here is the code I used, hacked directly from a tutorial then hacked up using some more notes I found elsewhere
Code:
#include <OneWire.h>

/* DS18b20 Temperature chip i/o */

OneWire  ds(22);  // on pin 22

void setup(void) {
  Serial.begin(9600);
}

void loop(void) {
  int HighByte, LowByte, TReading, SignBit, Tc_100, Whole, Fract;
  byte i;
  byte present = 0;
  byte data[12];
  byte addr[8];
 
  if ( !ds.search(addr)) {
    Serial.print("No more addresses.\n");
    ds.reset_search();
    delay(250);
    return;
  }
 
  Serial.print("R=");
  for( i = 0; i < 8; i++) {
    Serial.print(addr[i], HEX);
    Serial.print(" ");
  }

  if ( OneWire::crc8( addr, 7) != addr[7]) {
      Serial.print("CRC is not valid!\n");
      return;
  }
 
  if ( addr[0] != 0x28) {    //changed the value to 0x28 to recognize ds18b20
      Serial.print("Device is not a DS18B20 family device.\n");
      return;
  }

  // The DallasTemperature library can do all this work for you!

  ds.reset();
  ds.select(addr);
  ds.write(0x44,1);        // start conversion, with parasite power on at the end
 
  delay(1000);    // maybe 750ms is enough, maybe not
  // we might do a ds.depower() here, but the reset will take care of it.
 
  present = ds.reset();
  ds.select(addr);   
  ds.write(0xBE);        // Read Scratchpad

  Serial.print("P=");
  Serial.print(present,HEX);
  Serial.print(" ");
  for ( i = 0; i < 9; i++) {          // we need 9 bytes
    data[i] = ds.read();
    Serial.print(data[i], HEX);
    Serial.print(" ");
  }
  Serial.print(" CRC=");
  Serial.print( OneWire::crc8( data, 8), HEX);
  Serial.println();
  LowByte = data[0];
  HighByte = data[1];
  TReading = (HighByte << 8) + LowByte;
  SignBit = TReading & 0x8000;  // test most sig bit
  if (SignBit) // negative
  {
    TReading = (TReading ^ 0xffff) + 1; // 2's comp
  }
  Tc_100 = (6 * TReading) + TReading / 4;    // multiply by (100 * 0.0625) or 6.25

  Whole = Tc_100 / 100;  // separate off the whole and fractional portions
  Fract = Tc_100 % 100;


  if (SignBit) // If its negative
  {
    Serial.print("-");
  }
  Serial.print(Whole);
  Serial.print(".");
  if (Fract < 10)
  {
    Serial.print("0");
  }
  Serial.print(Fract);

  Serial.print("\n");
 
}
You will need to install the onewire library but that is easy enough to find with google, then just extract the zip into the correct location which in my case is /sketchbook/libraries

I think my next thing will be getting multiple temp sensors working hopefully that is easy. After that I want to try out those current sensors but that will involve a bit of soldering.

strider3700 12-07-10 02:53 PM
ok the second temp sensor is trivial. connect pin 1 and pin3 to ground on the arduino, connect pin 2 to pin 22 the same as the other sensor. I didn't even shut the arduino down or update the code. the serial monitor just started reading the second one as soon as it was hooked up.

right now they are reporting that my office is 17.93 or 17.52 C and they are 3 inches apart on the bread board. They are sensitive enough that if I hold my hand a couple inches above them the temperature starts climbing within 5 seconds. Now to order 25 more and track down a crap load of wire to do the entire house. This is too cool.

Next step I think will be getting this data logging and accessible online. If I track down some 3 wire or an extra cat 5 cable I'll try the current sensors out but that will take some non-coding work.

<edit> Ok I just ran across this page http://www.pachube.com/ it looks like it's tailor made for what I want to do here. I'll have to poke around and see if It will do what I want or if I need to roll my own. </edit>

<edit #2 > apparently pachube is not free if you want to store data for more then 30 days. the non free options range from really expensive to insanely expensive. $15/month to $900/month with 100% storage starting at $100/month... I'll play with it a bit for now but google apps is likely in my near future </edit #2>

strider3700 12-09-10 12:27 AM
Ok getting data logging going on even at the simplest level turned out to be stupidly difficult for me. I've only been programming for a couple of decades now but I don't think I've ever touched the serial port so I decided to just grab an example someone else did and go with that.

My initial goal is just to get it writing every output to a local file. Nothing hard at all right?

Well I first tried a simple batch script that just monitored the serial port but that got a lot of corrupted crap or dropped data. So I figured since the arduino is using python lets just find a python script and do that. Well 4 hours and 5 examples later it wasn't working correctly. Once again lots of errors and corrupted data. I'm also not a python guy so debugging not knowing the language was a nightmare.

So I went to java which isn't my day to day but I've done a bunch of and it's damn close to c# which I mostly do these days. Then I figured since I have a java example lets learn eclipse, I've heard good things about it. Well it's pretty and shiny but it took me awhile to figure out how to add a jar to the project to make the rxtx library work. Anyways here's the code I found after I changed the port to be correct
Code:
package datalogger;

        import java.io.InputStream;
        import java.io.OutputStream;
        import gnu.io.CommPortIdentifier;
        import gnu.io.SerialPort;
        import gnu.io.SerialPortEvent;
        import gnu.io.SerialPortEventListener;
        import java.util.Enumeration;

        public class SerialTest implements SerialPortEventListener {
                SerialPort serialPort;
                /** The port we're normally going to use. */
                private static final String PORT_NAMES[] = {
                                                                "/dev/ttyUSB0", // Linux
                                                                };
                /** Buffered input stream from the port */
                private InputStream input;
                /** The output stream to the port */
                private OutputStream output;
                /** Milliseconds to block while waiting for port open */
                private static final int TIME_OUT = 2000;
                /** Default bits per second for COM port. */
                private static final int DATA_RATE = 9600;

                public void initialize() {
                        CommPortIdentifier portId = null;
                        Enumeration portEnum = CommPortIdentifier.getPortIdentifiers();

                        // iterate through, looking for the port
                        while (portEnum.hasMoreElements()) {
                                CommPortIdentifier currPortId = (CommPortIdentifier) portEnum.nextElement();
                                for (String portName : PORT_NAMES) {
                                        if (currPortId.getName().equals(portName)) {
                                                portId = currPortId;
                                                break;
                                        }
                                }
                        }

                        if (portId == null) {
                                System.out.println("Could not find COM port.");
                                return;
                        }

                        try {
                                // open serial port, and use class name for the appName.
                                serialPort = (SerialPort) portId.open(this.getClass().getName(),
                                                TIME_OUT);

                                // set port parameters
                                serialPort.setSerialPortParams(DATA_RATE,
                                                SerialPort.DATABITS_8,
                                                SerialPort.STOPBITS_1,
                                                SerialPort.PARITY_NONE);

                                // open the streams
                                input = serialPort.getInputStream();
                                output = serialPort.getOutputStream();

                                // add event listeners
                                serialPort.addEventListener(this);
                                serialPort.notifyOnDataAvailable(true);
                        } catch (Exception e) {
                                System.err.println(e.toString());
                        }
                }

                /**
                * This should be called when you stop using the port.
                * This will prevent port locking on platforms like Linux.
                */
                public synchronized void close() {
                        if (serialPort != null) {
                                serialPort.removeEventListener();
                                serialPort.close();
                        }
                }

                /**
                * Handle an event on the serial port. Read the data and print it.
                */
                public synchronized void serialEvent(SerialPortEvent oEvent) {
                        if (oEvent.getEventType() == SerialPortEvent.DATA_AVAILABLE) {
                                try {
                                        int available = input.available();
                                        byte chunk[] = new byte[available];
                                        input.read(chunk, 0, available);

                                        // Displayed results are codepage dependent
                                        System.out.print(new String(chunk));
                                } catch (Exception e) {
                                        System.err.println(e.toString());
                                }
                        }
                        // Ignore all the other eventTypes, but you should consider the other ones.
                }
        public static void main(String[] args) {
                SerialTest main = new SerialTest();
                main.initialize();
                System.out.println("Started");
        }

}
I got it from here. Arduino playground - Java
I like that it's event driven it totally gets rid of all those issues I had with the other attempts. Anyways the hard part was adding the rxtx jar recognized in eclipse. I followed this partially (it's for windows I'm on linux) Using RXTX In Eclipse - Rxtx. That got it working so I can now go back to some old school reading and writing to disk...

I think the major Issue I've had with getting this part working is there are hundreds of ways to do it and none of them are amazingly well documented. The arduino people of course don't recommend one method over another and google will net you thousands of partial chunks of code pointing you in the right direction but none of them are plug and play.

Daox 12-09-10 06:08 AM
While I haven't taken the time to really dig into your posts on this yet, please keep posting as it does look quite interesting!

strider3700 12-09-10 12:06 PM
1 Attachment(s)
I have to say I love this thing so far. Last night it logged temperatures in my office every minute to a csv file. This morning I threw it into a spreadsheet and charted it.

http://ecorenovator.org/forum/attach...1&d=1291917405

The spikes every 15 minutes or so are the circulating fan for the furnace kicking on. The heat pump is not actively heating the fan is just helping to move the hot air from the woodstove around. I was never sure if it was actually effective but it is obviously pushing a bit of hot air in here.

THat last spike is also something I've noticed manually watching the sensor data. Thats me coming into the office and sitting down 18 inches from the sensor without a shirt on. People release a lot of heat.

Tonights goal will be to get the java program parsing the incoming stream better. Right now I just raw right it to a file and that is handling the new lines and putting the chunks together. I want to get it actually parsing first of all complete lines then split those lines up so that I can start doing averaging and so on inside of the program.

Also Sensor 282895C50200009B is a really crappy label. That is one of my complaints with the one wire system so far. I will need to install each sensor record it's ID in HEX then label it so that when I have 20 of them out there I know which room has which sensor so that I can apply a label to them. If not it's going to be a game of put your hand on it and sit here watching for the temp to go up then apply the label.

strider3700 12-10-10 01:17 PM
To not screw up my simple logging and get an easy pretty graph I had turned off the second sensor reading on the arduino rather then parse it on the java program. Last night I turned it back on and discovered I had an issue. The Event that fires saying data is on the serial port grab it would happen quicker then the java program could parse the first event so basically I'd get the entire first sensor in except for the new line character and then the second sensor would partially send garbling the results. I went to bed thinking that this would suck to handle, I'd need a second thread to process data and one to just grab the data. This morning I woke up realizing it's far far easier to just add a delay between sending each sensors data. so the send part on my arduino now looks like this.
Code:
printData(s1);
delay(100);
printData(s2);
It's a hack but it works. I'll play around with the delay to see how short I can get it without causing issues. I don't want to take 3 seconds to read in 30 sensors worth of data if possible....

<edit> A delay of 30 between each sensor write is good enough to allow me to output 100 sensors worth of readings. I've moved the delay into the print data function on the arduino. </edit>

strider3700 12-10-10 01:32 PM
1 Attachment(s)
And last nights office temps

You can clearly see the effects of me closing the office door right at the start and then opening it in the morning right at the end. THe gradual middle of the night climb is from me getting the fire going better.

http://ecorenovator.org/forum/attach...1&d=1292009468

strider3700 12-10-10 01:47 PM
My last edit reminded me. I started using the dallas temperatures library on the arduino so it's code is very different and much cleaner. It's still in bodge it together to see if it works stage though so it's far from pretty but here is the complete arduino code as it currently stands.

Code:
#include <OneWire.h>
#include <DallasTemperature.h>

// Data wire is plugged into port 22 on the Arduino
#define ONE_WIRE_BUS 22
#define TEMPERATURE_PRECISION 9
#define delayTime 2000    // the delay before restarting the loop

// Setup a oneWire instance to communicate with any OneWire devices (not just Maxim/Dallas temperature ICs)
OneWire oneWire(ONE_WIRE_BUS);

// Pass our oneWire reference to Dallas Temperature.
DallasTemperature sensors(&oneWire);

// arrays to hold device addresses
DeviceAddress s1, s2;

void setup(void)
{
  // start serial port
  Serial.begin(9600);
 
  // Start up the library
  sensors.begin();

  //printStatus();

  // assign address manually.  the addresses below will beed to be changed
  // to valid device addresses on your bus.  device address can be retrieved
  // by using either oneWire.search(deviceAddress) or individually via
  // sensors.getAddress(deviceAddress, index)
  //insideThermometer = { 0x28, 0x1D, 0x39, 0x31, 0x2, 0x0, 0x0, 0xF0 };
  //outsideThermometer  = { 0x28, 0x3F, 0x1C, 0x31, 0x2, 0x0, 0x0, 0x2 };

  // search for devices on the bus and assign based on an index.  ideally,
  // you would do this to initially discover addresses on the bus and then
  // use those addresses and manually assign them (see above) once you know
  // the devices on your bus (and assuming they don't change).
  //
  // method 1: by index
  if (!sensors.getAddress(s1, 0)) Serial.println("Unable to find address for Device 0");
  if (!sensors.getAddress(s2, 1)) Serial.println("Unable to find address for Device 1");

  // method 2: search()
  // search() looks for the next device. Returns 1 if a new address has been
  // returned. A zero might mean that the bus is shorted, there are no devices,
  // or you have already retrieved all of them.  It might be a good idea to
  // check the CRC to make sure you didn't get garbage.  The order is
  // deterministic. You will always get the same devices in the same order
  //
  // Must be called before search()
  //oneWire.reset_search();
  // assigns the first address found to insideThermometer
  //if (!oneWire.search(insideThermometer)) Serial.println("Unable to find address for insideThermometer");
  // assigns the seconds address found to outsideThermometer
  //if (!oneWire.search(outsideThermometer)) Serial.println("Unable to find address for outsideThermometer");

 
  // set the resolution to 12 bit
  sensors.setResolution(s1, 12);
  sensors.setResolution(s2, 12);

 
}

// function to print a device address
void printAddress(DeviceAddress deviceAddress)
{
  for (uint8_t i = 0; i < 8; i++)
  {
    // zero pad the address if necessary
    if (deviceAddress[i] < 16) Serial.print("0");
    Serial.print(deviceAddress[i], HEX);
  }
}

void printStatus(void)
{
  // locate devices on the bus
  Serial.print("Locating devices...");
  Serial.print("Found ");
  Serial.print(sensors.getDeviceCount(), DEC);
  Serial.println(" devices.");

  // report parasite power requirements
  Serial.print("Parasite power is: ");
  if (sensors.isParasitePowerMode()) Serial.println("ON");
  else Serial.println("OFF");
 
  // show the addresses we found on the bus
  Serial.print("Device 0 Address: ");
  printAddress(s1);
  Serial.println();

  Serial.print("Device 1 Address: ");
  printAddress(s2);
  Serial.println();
 
  // print the device resolutions
  Serial.print("Device 0 Resolution: ");
  Serial.print(sensors.getResolution(s1), DEC);
  Serial.println();

  Serial.print("Device 1 Resolution: ");
  Serial.print(sensors.getResolution(s2), DEC);
  Serial.println();
}
// function to print the temperature for a device
void printTemperature(DeviceAddress deviceAddress)
{
  float tempC = sensors.getTempC(deviceAddress);
  //Serial.print("Temp C: ");
  Serial.print(tempC);
}

// function to print a device's resolution
void printResolution(DeviceAddress deviceAddress)
{
  Serial.print("Resolution: ");
  Serial.print(sensors.getResolution(deviceAddress));
  Serial.println();   
}

// main function to print information about a device
void printData(DeviceAddress deviceAddress)
{
  //Serial.print("Device Address: ");
  printAddress(deviceAddress);
  Serial.print(",");
  printTemperature(deviceAddress);
  Serial.println();
  delay(30);
}

void loop(void)
{
  // call sensors.requestTemperatures() to issue a global temperature
  // request to all devices on the bus
  //Serial.print("Requesting temperatures...");
  sensors.requestTemperatures();
  //Serial.println("DONE");

  // print the device information
  printData(s1);
  printData(s2);
  delay(delayTime);
}

AC_Hacker 12-10-10 09:34 PM
Quote:
Originally Posted by strider3700 (Post 10198)
You can clearly see the effects of me closing the office door right at the start and then opening it in the morning right at the end. THe gradual middle of the night climb is from me getting the fire going better.[/IMG]
I confess that I haven't been following this thread from the beginning, so pardon if I've missed something...


I understand the trend lines, but what causes the short-duration periodic swings?

-AC_Hacker

strider3700 12-10-10 10:32 PM
I have the fan in the forced air system set to circulate so it comes on for a few minutes every 15 minutes or so. the room with the sensor is easily the coldest in the house being the basement north west corner so slightly warmer air gets blown in when that happens. if the air is warm enough the room gradually climbs as happened last night. If the air isn't the room gradually drops like happened in the first graph.

I picked up 100 meters of cat 3 today and will put the second sensor in the room with the wood stove if I get a chance tonight. It'll be nice to see multiple temps logged and see how the rooms are different.

AC_Hacker 12-11-10 12:37 AM
Quote:
Originally Posted by strider3700 (Post 10215)
I have the fan in the forced air system set to circulate so it comes on for a few minutes...
OK, that explains everything.

* * *

Say, I'm only marginally able to tweak Arduino code, and what you are doing is right in line with something I have wanted to do for a couple of years...

As you may know, I'm doing the Homemade HeatPump Manifesto thread, and I have wanted to build a vertical multi-sensor data logger to monitor temperatures of the earth at various depths in my GSHP loop field.

My idea is to get a fifteen foot length of pipe, copper is ok, PVC would work too, and put sensors every two feet or so, and fill the pipe up with tar or plastic or something and bury it, putting a data cable from the pipe and sensors, into a foot deep trench, and run it into the house.

Then I would monitor the temps periodically like maybe once an hour or once a day... for several years.

I understand that 1-wire sensors are uniquely numbered, and quite accurate, and for design simplicity, I would run them off of three wires, Data, Vcc, and Ground. Screw the timeing issues, wire is cheap.

So, do you think you could help me out with code to read several 1-wire therrmal sensors?

I'm guessing it's nothing much more than cut & paste & paste & paste & paste, etc.

Tell me, am I over-simplifying?

Does your code include an initialization section that polls and records the unique identifiers of all sensors?

Thanks,

-AC_Hacker

strider3700 12-11-10 12:55 AM
Just FYI the ds18b20 sensors will not work over 100 meters of cat 3 28 gauge 2 pair in either parasitic or standard mode. I'm not shocked and don't need them to anyways but wanted to test. It does work just fine in parasitic for 10 meters though. My longest is about 25 meters but I won't be setting that test up tonight.

strider3700 12-11-10 01:14 AM
Quote:
Originally Posted by AC_Hacker (Post 10218)

So, do you think you could help me out with code to read several 1-wire therrmal sensors?
Not a problem. Before I go to bed I'm hoping to have the logging to csv of multiple sensors working. The reading is already there.

Your basic steps are -

first use a breadboard and wire up 1 sensor

Here's a nice page explaining it. Arduino and DS18B20 - 1-wire digital thermometer | Ogalik

on the arduino you'll see VCC and gnd so you plug them in. Pick any of the digital pins you just need to know the number.

Then load the arduino IDE and install the dallas temps and 1 wire library - it's pretty trivial and I can point it out if needed when the time comes, basically just put the correct file from a downloaded zip in the right space.

Then if you grab my code a couple of posts back paste it into the arduino IDE change the pin number at the top to whatever you used and hit compile it should compile with no errors. click upload and then fire up the serial monitor in the arduino IDE and it should tell you the sensors ID and the temperature.

record the ID and label the sensor, repeat till you have ID's for all of them

wire two of them up like in that page I posted and restart the serial monitor. The code will work fine for two with no changes as it currently is. Soon I'll make it work for any number (up to 64 is the limit I think) of sensors without any changes and you'll be good to go.

That will give you data being dumped into the arduino serial monitor. When the java part of the program is finished I'll post it up and you can dump the data to a csv. Or you can grab the code as it currently stands and track down the rxtx library and build it yourself.

As I go whatever I create while working on this project I'll post on here and anyone can take and do whatever with it. Just be forewarned I'm posting where I'm at every couple of days and much of it is far from optimal. I'll hopefully get it pretty by the end but if you grab something now just be aware that it's a very early stage work in progress/proof of concept.

I am hoping that the java code will be good enough in the end to be useful for others that want basic data logging via an arduino. Just have the arduino return the sensor ID and a value and the java program will put that to a csv file.

AC_Hacker 12-11-10 02:08 AM
Thanks, most of it is pretty clear.

Tomorrow, I'll dig up my arduino board and give it a try.

-Regards,

-AC_Hacker

strider3700 12-11-10 04:04 AM
Ok I spent some time wiring up a second temp sensor and installing it in my rec room which has the woodstove. It was an easy/quiet wiring job to do in the middle of the night and it should be interesting to see how the temps work out in my office vs in there.

Then I decided to get down to work on the java side to get logging that in a nice easy to use method. I don't want to datalog every second or two's data but I don't want the inaccuracy associated with only collecting data every 1,2,5... whatever minutes. So I figured I'll collect data every second and average it for whatever time period becomes useful in the future.

So I decided first of all I need a new java class to create a sensor object for each sensor. it gives me the basics, set the name, id, get the name, id. also it has a Value stored in it. I can reset that, and add to it. each time I add to it I add one to the count and when I ask for the average I just take the value/count and return that. Pretty simple.
Code:
package datalogger;

public class Sensor {
       
        private String sensorID;
    private String sensorName;
    private float value;
    private int averageCount;
   
    Sensor(String inID,String inName)
    {
    sensorID = inID;
    sensorName = inName;
    value =0;
    averageCount = 0;
    }
   
    public String getID(){
            return sensorID;
    }
    public String getID(String inName){
            return sensorID;
    }
   
    public String getName(){
            return sensorName;
    }
   
    public String getName(String inID){
            return sensorName;
    }
       
    public void setName(String inName)
    {
            sensorName = inName;
    }
   
    public void setID(String inID){
            sensorID = inID;
    }
    public void resetValue()
    {
            value =0;
            averageCount =0;
    }
   
    public void addValueToAverage(float inValue)
    {
    value += inValue;
    averageCount++;
    }
   
    public String returnAverage()
    {
      float average = value/averageCount;
      return (String.valueOf(average));
    }     
   
}
The other class got some more functions calling and using the sensor class but nothing special. I'm thinking I should really create a new class to handle the data manipulation that will just call the serialtest and let it handle getting the data from the serial port then hand it over to this other class to deal with. I won't be working on this much if at all tomorrow though. I also somewhere need to add a timer that when the timer goes off will walk through the list of sensors and get their averages then output them to the CSV as 1 line to make for easy comparison graphs.

I was also thinking that long term I should make the program look at a simple configuration file to get the sensor ID's and names to make it easier for non programmers to use. As if I don't have enough to do...

strider3700 12-13-10 03:38 AM
Ok I've spent too much time on this lately but it's now doing the following.

The arduino dumps back all of the sensor data every 2 seconds with a 30 millisecond delay between each.

The java program collects the data and every 60 seconds prints the average for each sensor to a csv file the clears the average so we're getting a minute by minute recording. The format going into the csv file is Date,Time,S1,S2.... with the first line being a Header which loads the "common name" for the sensor so in my case, office, Rec room. I don't know if there is a guarantee that s1 will always be office and s2 always rec room. The program will always put the values to the correct sensor name however and it won't flip part way through a run. Worst case you run the program and it lists them office, rec room and then you stop the program restart it and it will continue appending but flipped so recroom, office. The Header will be rewritten if you were to do that.... Common names are currently hardcoded. Long term none of that should matter because I want to move away from dumb csv's and into some form of database.

anyways right now it's running collecting from the two different rooms and in the morning I can generate a new chart, this one with labels and two lines... That's enough to keep me happy for awhile data wise so when I return to this project it will be to get current readings working.

Now the code if anyone is interested. It's still rough... The arduino is the same as last time I posted I believe.

Logger.java the new main class
Code:
package datalogger;

import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Timer;

public class Logger {

        private static Date today;
        private static SimpleDateFormat  formatter = new SimpleDateFormat("yyyy/MM/dd,HH:mm");
        public static List<Sensor> sensorList=new ArrayList<Sensor>();
        private static SerialHandling handler ;
        private static Timer timer;
        public  static void main(String[] args) {
                loadSensorList();
                timer = new Timer();
                setTimer();
               
                handler  = new SerialHandling();
                handler.initialize();
               
        }
               
       
        private static void setTimer(){
                timer.scheduleAtFixedRate(
                                new writeData(),  // calls the writeData class run method to write the data to csv
                                0,                                  // start right away
                                60000            // run every 60 seconds
                                //5000                          // run every 5 seconds for testing       
                );
        }
        public static void parseText(String line)
        {
                //System.out.println(line);
                if (line.contains(","))
                {
                        String[] split = line.split(",");                                       
                        String sensor = split[0];
                        String value = split[1];

                        averageValues(sensor,value);

                        String name = getSensorName(sensor);                                       
                        today = new Date();
                        String time = formatter.format(today);
                        //System.out.println(name+","+time+","+value);
                        System.out.println(time+","+name+","+value);
                        //writeText(time+" "+name+" "+ value+newline);
                        //writeText(name+","+time+","+value+newline);
                }
        }
        // fills the List with the Sensors.  Currently hardcoded ID's
        public static void loadSensorList()       
        {               
                Sensor office = new Sensor("282895C50200009B","office");
                Sensor recroom = new Sensor("28BF90C50200007D","recroom");
                sensorList.add(office);
                sensorList.add(recroom);               
        }

        public static int getSensorCount()
        {
                return sensorList.size();
        }
       
       
        // find the sensor based on the ID
        private static Sensor getSensor(String inID)
        {
                int i =0;
                boolean done = false;
                Sensor temp = new Sensor("default","default");
                while ( i < sensorList.size() && done == false)
                {
                        temp = sensorList.get(i);

                        if (inID.equals(temp.getID()))
                        {
                                done = true;                                               
                        }               
                        i++;
                }
                return temp;
        }

        // returns the sensors common Name if in the list else the sensor ID is used as the name
        private static String getSensorName(String inID){

                String name = inID; //default to the ID if not found
                Sensor temp = getSensor(inID);                               
                name = temp.getName();
                return name;
        }
       
        private static void averageValues(String id, String value){

                Sensor s = getSensor(id);
                try
                {
                        float f = Float.valueOf(value.trim()).floatValue();
                        s.addValueToAverage(f);
                }
                catch (NumberFormatException nfe)
                {
                        System.out.println("NumberFormatException: " + nfe.getMessage());
                }

                //  System.out.println(id+" value "+value+" average "+s.returnAverage());

        }
}
WriteData.java - the timer that writes to the csv every minute
Code:
package datalogger;

import java.io.BufferedWriter;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.TimerTask;


// the Timer task the calls the outputting to the CSV file
public class writeData extends TimerTask {

        private static Date today;
        private static SimpleDateFormat  formatter = new SimpleDateFormat("yyyy/MM/dd,HH:mm");
        private boolean first = true;
       
        public void run() {
                //System.out.println("Generating csv");
                String row = buildRow();
                if (row.contains("NaN")==false) // done to stop that initial hit of empty from being written I could also delay
                {                                                                // the timer by 60 seconds but this always works no matter what the timer timing
                        writeText(row);
                }               
        }
       
        //builds the row to output to CSV file. It hardcodes the header line in the first time called. 
        private String buildRow(){
               
                //row = Date,Time,s1,s2,...
                String row = "";
                // if the first line then dump in the header
                if (first == true)
                {
                        row = "Date,Time";
                        for (int i=0;i< Logger.getSensorCount() ;i++)
                        {
                                row+=","+Logger.sensorList.get(i).getName();
                        }
                        first = false;
                        writeText(row);
                        row ="";
                }
               
                today = new Date();
                String time = formatter.format(today);
                row += time;  // add the time to the row
               
                // now add all of the average values in the sensors
                for (int i=0;i< Logger.getSensorCount() ;i++)
                {
                       
                        row+=","+Logger.sensorList.get(i).returnAverage();        //get the average
                        Logger.sensorList.get(i).resetValue();  // clear the averages
                }
               
                //System.out.println("writing -"+row);
                return row;
        }
       
        // for writing the text strings to the file       
        public synchronized void writeText(String inString){
                String strFilePath = "/home/strider/Logs/tempData.csv";
                String newline = System.getProperty("line.separator");
                try
                {
                        Writer output = new BufferedWriter(new FileWriter(strFilePath,true));

                        //FileWriter always assumes default encoding is OK!
                        output.write( inString+newline);
                        output.close();
                }
                catch(FileNotFoundException ex)
                {
                        System.out.println("FileNotFoundException : " + ex);
                }
                catch(IOException ioe)
                {
                        System.out.println("IOException : " + ioe);
                }
        }       
}

strider3700 12-13-10 03:39 AM
stupid maximum length rule ;)


SerialHandler.java - handles the data on the serial port and passes it formatted to Logger
Code:
package datalogger;

import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import gnu.io.CommPortIdentifier;
import gnu.io.SerialPort;
import gnu.io.SerialPortEvent;
import gnu.io.SerialPortEventListener;
import java.util.Enumeration;



public class SerialHandling implements SerialPortEventListener {
        SerialPort serialPort;
        /** The port we're normally going to use. */
        private static final String PORT_NAMES[] = {
                "/dev/ttyUSB0", // Linux
        };
        /** Buffered input stream from the port */
        private InputStream input;
        /** The output stream to the port */
        private OutputStream output;
        /** Milliseconds to block while waiting for port open */
        private static final int TIME_OUT = 2000;
        /** Default bits per second for COM port. */
        private static final int DATA_RATE = 9600;               
        private String line = new String();

        public void initialize() {                       
                CommPortIdentifier portId = null;
                Enumeration portEnum = CommPortIdentifier.getPortIdentifiers();

                // iterate through, looking for the port
                while (portEnum.hasMoreElements()) {
                        CommPortIdentifier currPortId = (CommPortIdentifier) portEnum.nextElement();
                        for (String portName : PORT_NAMES) {
                                if (currPortId.getName().equals(portName)) {
                                        portId = currPortId;
                                        break;
                                }
                        }
                }

                if (portId == null) {
                        System.out.println("Could not find COM port.");
                        return;
                }

                try {
                        // open serial port, and use class name for the appName.
                        serialPort = (SerialPort) portId.open(this.getClass().getName(),
                                        TIME_OUT);

                        // set port parameters
                        serialPort.setSerialPortParams(DATA_RATE,
                                        SerialPort.DATABITS_8,
                                        SerialPort.STOPBITS_1,
                                        SerialPort.PARITY_NONE);

                        // open the streams
                        input = serialPort.getInputStream();
                        output = serialPort.getOutputStream();

                        // add event listeners
                        serialPort.addEventListener(this);
                        serialPort.notifyOnDataAvailable(true);
                } catch (Exception e) {
                        System.err.println(e.toString());
                }
        }

        /**
        * This should be called when you stop using the port.
        * This will prevent port locking on platforms like Linux.
        */
        public synchronized void close() {
                if (serialPort != null) {
                        serialPort.removeEventListener();
                        serialPort.close();
                }
        }

        /**
        * Handle an event on the serial port. Read the data and print it.
        */
        public synchronized void serialEvent(SerialPortEvent oEvent) {

                if (oEvent.getEventType() == SerialPortEvent.DATA_AVAILABLE) {
                        try {
                                int available = input.available();
                                byte chunk[] = new byte[available];
                                input.read(chunk, 0, available);

                                // Displayed results are codepage dependent                                                               
                                //System.out.print(new String(chunk));

                                //writeText(chunk);
                                parseText(new String(chunk));
                                //System.out.println(line);

                        } catch (Exception e) {
                                System.err.println(e.toString());
                        }
                }
                // Ignore all the other eventTypes, but you should consider the other ones.
        }

        public synchronized void parseText(String chunk)
        {
                String newline = System.getProperty("line.separator");
                line+= chunk;

                if (line.contains("\\n") || line.contains("\\r") || line.contains(newline))
                {
                        line = removeReturns(line);
                        if (line.contains("Unable to find address for Device"))
                        {
                                System.out.println(" Connection failed retry"); // sometimes the arduino fails to recognize the sensor ID's 
                        }                       
                        Logger.parseText(line);                       
                        line =""; // line is dealt with so clear it
                }

        }
       
        // removes the multiple return characters from the incoming line.
        private String removeReturns(String inLine){
                String newline = System.getProperty("line.separator");

                inLine = inLine.replaceAll(newline,"");
                inLine = inLine.replaceAll("\\n", "");  // replacing the newline characters with blank.
                inLine = inLine.replaceAll("\\r", "");  // It is getting both \n and \r so both need handled.       

                return(inLine);

        }

        // for writing the raw byte array to file
        public synchronized void writeText(byte[] inChunk){
                String strFilePath = "/home/strider/Logs/tempData.csv";

                try
                {
                        FileOutputStream fos = new FileOutputStream(strFilePath,true);  //true to append

                        /*
                        * To write byte array to a file, use
                        * void write(byte[] bArray) method of Java FileOutputStream class.
                        *
                        * This method writes given byte array to a file.
                        */

                        fos.write(inChunk);

                        /*
                        * Close FileOutputStream using,
                        * void close() method of Java FileOutputStream class.
                        *
                        */

                        fos.close();

                }
                catch(FileNotFoundException ex)
                {
                        System.out.println("FileNotFoundException : " + ex);
                }
                catch(IOException ioe)
                {
                        System.out.println("IOException : " + ioe);
                }
        }       
}
Sensor.java - I think it's the same as before, basically everything to do with the sensor objects.
Code:
package datalogger;

public class Sensor {
       
        private String sensorID;
    private String sensorName;
    private float value;
    private int averageCount;
   
    Sensor(String inID,String inName)
    {
    sensorID = inID;
    sensorName = inName;
    value =0;
    averageCount = 0;
    }
   
    public String getID(){
            return sensorID;
    }
    public String getID(String inName){
            return sensorID;
    }
   
    public String getName(){
            return sensorName;
    }
   
    public String getName(String inID){
            return sensorName;
    }
       
    public void setName(String inName)
    {
            sensorName = inName;
    }
   
    public void setID(String inID){
            sensorID = inID;
    }
    public void resetValue()
    {
            value =0;
            averageCount =0;
    }
   
    public void addValueToAverage(float inValue)
    {
    value += inValue;
    averageCount++;
    }
   
    public String returnAverage()
    {
      float average = value/averageCount;
      return (String.valueOf(average));
    }         
}
If anyone wants/needs anything or has any questions just let me know.

strider3700 12-13-10 01:20 PM
1 Attachment(s)
And this mornings graph verifying that everything is working correctly.
http://ecorenovator.org/forum/attach...1&d=1292267888

It appears that taking the average of the every 2 second readings every minute has partially flattened the spikes from the circulating fan kicking on.

strider3700 12-19-10 05:10 PM
Well 6 days later everything is still working although I have found a limitation to my current logging technique. Saving a row every minute in a CSV file works just fine. The file is now 323.1kb which works out to about 20 MB a year which is nothing in my mind. It is however 8600 or so rows and open office has started to gak on it. Opening the file now takes 20 seconds and creating a chart takes close to 2 minutes. If I try to do any smoothing of the data it takes over 10 minutes. Simply resizing the chart when it's produced grinds on the new very fast quad core 8 gig's of ram machine.

This isn't shocking there is a reason real databases don't store everything in csv files. Things are really busy here but I may need to start on the link to the google apps DB or a local DB really soon. I want to learn the google apps version and get that data up for everyone to see but local is free, easy and not too difficult.

Another thing I also need to do is take a look at how much space is on the arduino it's self and see if I can get the code to store the data for awhile if the serial port isn't available. I had to reboot once and lost 4 minutes of data, which is far from critical but I like my data to be complete.

strider3700 12-26-10 01:55 PM
I got back on the project yesterday after just letting the two temps sensors collect data for a week or so.

During that week I had the serial port "vanish" once and after unplugging the arduino and plugging it back in to the USB port it went back to working just fine. I'll have to see if I can figure out what caused that as it has happened twice. All data being collected is gone when that happens of course.

I've decided at this point I'm going to change my logging to CSV files to be a new file daily. so at midnight the file name being written to will change to datalog-ddmmyyyy.csv This is of course still just a temporary datastorage but I will probably leave it in place long term when I'm done with the online logging as well. The code for that should be easy so maybe tonight I'll knock that out.

Much more interesting then my "issues" however is the progress. Yesterday I had the soldering iron out and decided to attach a chunk of the 2 pair to one of the current sensors. I then took my homemade extension cord for the wood stove which has those screw together plugs apart and ran one of the wires through the hole in the current sensor. Wiring to the arduino was simple, 5v to vcc, gnd to gnd, en to gnd(this could be switched on the arduino but power usage at this scale isn't import enough to use a pin), data to one of the analog in pins on the ardunio(0 in my case)

then in the arduino sketch I added a small section which basically just calls analogread(0); This reads the current voltage at pin 0 between slightly over 0 and slightly below 5V and turns it into an integer between 0 and 1024 with each working out to roughly 0.0049V. It turns out the current sensor returns roughly 509 when no current is flowing through it. So in the arduino code I just subtract 509 from the value returned and just get anything above or below that.

Last night I had a hell of a time getting any good readings and it was making little sense until I graphed it and went to bed to sleep on the issue. I was always getting 506-515 even when I had a 1500 watt heater plugged in that was using in the 1450 range according to my wattmeter. I was taking about 100 readings per second and averaging trying to get rid of noise on the line thinking it was noise....

This just goes to show that I'm a programmer not an electronics guy. AC is a signwave and it will go both positive and negative in one cycle. The current also seems to be going in a wave as far as I can tell and I really really didn't want an average I wanted to know the peak value each second. Realizing this while sleeping last night, this morning I got up and changed my reading code to loop through 1000 times with a delay of 1 ms each loop and take the value each time. I then took the absolute value of it and checked if it was higher then the previously recorded max. if so max became that reading, continue the loop. At the end of the loop print out the max and reset it to zero.

This works great and I'm now able to graph with only minor noise jitters the hater turning on the fan, to high, to med to off to high.... with the values being anywhere from zero to 71 and it to the naked eye appearing pretty linear. It appears to have a "resolution" of about 7 watts. Anything below that vanishes in noise. I also get the occasional spike of noise in the 7 watt range.

If you want really sensitive down to the watt 100 times per second readings my approach isn't the way to go but for having a good idea how much power something large is using this works out great and it cost me $5 for 1 sensor.

Now my next step is to get the values converted to watts for logging since that is what I care about. That should just be some math on the arduino. I also think I may change the datalogging program to handle current sensors a little differently. A minutely average may not be accurate enough.

That is all just a bit of code. The next interesting thing is figuring out how to get a 220V wattage using two of these sensors on each line running into the hotwater heater. That was the real goal of the current readings. For now I'll just wire them up and get the readings back from both sensors so the rest should just be math. Do I just sum the two?

All code for both sections will be available when I get it a bit less hacked together.

strider3700 12-26-10 03:37 PM
While researching if summing two 110 legs of a 220 V circuit will give me the value I want I discovered RMS voltage. This is why I love doing projects from scratch myself, I learn a crap load in areas I know very little about.

Anyways when the kids are napping I'm going to give RMS a try and see if I can get that math working out. Right now I have no exact mains voltage reading so I'll have to hardcode based on what the wattmeter reads to calculate my wattage used.

strider3700 12-27-10 02:35 AM
well the RMS calculations didn't work out. my base reading from the sensor with no current is 1/2 vcc which gets translated to roughly 507. Hooking up the heater I get up to 570 on high which the kill a watt reports to be 1300 watts. The RMS calculation climbed to 509 maybe 510 doing the math in a spreadsheet using the current sensor datasheets estimate of approximately .023 mV/amp The peak readings was off by about 15% adjusting the math to 0.026 mV/amp I get within 2% for all of my readings done above 500 watts. The values below 500 watts are 12, 17,26,53 and they are off by 47%,75%,47%,18%.

At this point I'm going to go with this method and just the higher adjusted value. The water heater is closer to 5000 watts then 50 watts so hopefully things stay close up there. I'd love to get a real instantaneous reading to know how close I am on the tank but I don't have the equipment to do it.

Another concern I have is the watt meter tells me I have 119 V, when I kick the heater on full the voltage drops to 114 V. I'm assuming the same will happen with the hotwater heater however it won't be running on a close to overloaded circuit like the portable heater was. I'll probably just guess at the voltage on the line as being 230 and live with that. Now that I think about it I suppose I could read it directly using the multimeter one day when the heater is running.

hamsterpower 12-27-10 07:39 AM
Quote:
Originally Posted by strider3700 (Post 10594)

I've decided at this point I'm going to change my logging to CSV files to be a new file daily. so at midnight the file name being written to will change to datalog-ddmmyyyy.csv This is of course still just a temporary datastorage but I will probably leave it in place long term when I'm done with the online logging as well. The code for that should be easy so maybe tonight I'll knock that out.
Just to keep your files in the order they were collected, I would write the file name "datalog-yyyymmdd.csv". Otherwise great project.

Daox 12-27-10 08:05 AM
Wooo current data logging, very interesting. What current sensors are you using btw?

strider3700 12-27-10 12:13 PM
good point hamster.


Doax I'm using 1 of these Linear BiPolar Hall Effect Current Sensor +/- 80 Amps. I picked up 4 and with shipping they cost less then $30 total if I remember correctly.

I have 2 more that will be going to the hotwater heater probably tonight. And that leaves me 1 more which I haven't decided what to do with yet. I may pick up a couple of the larger ones with the 1/2" openings to monitor power usage on the heat pump. It's got some thick cables running to it.


Last night I took my test cord back and plugged the woodstove in. It only took a couple of minutes with the kilawatt meter to determine voltage, and watts used by the fan then tweak the code in the arduino so that it matches or very close to If I remember kilawatt readings are 42W low, 74W high and the arduino now reads 43W low 74 high by changing the 0.26mV/amp to 0.40 mV/amp. This gave pretty accurate readings across the board from 16-74W but would be way way too low above 500W.

Since you don't move the sensors around easily and my loads are going to be pretty consistent hardcoding that adjustment into a function used just for the woodstove works just fine for me. If I was making this completely generic I could do a table of voltage-conversion factors.

strider3700 12-28-10 11:23 PM
An update - I'm stressed beyond belief which sucks but the good news is I tinker/code to relax so this morning I had my 2 year old measure out two length of 2 cable and I soldered on two more current sensors then hooked them up to the hotwater tank. The run is far from permanent or professional looking(do pro's stick wires to the ceiling with ducttape?) but it is hooked up and "working"

Here is my hotwater usage since a few minutes after noon.

http://lh5.ggpht.com/_w1_9lMbNbxo/TR...reenshot-4.png

The first big flat usage is my shower then my wifes, both extra long and hot for the sake of scientific discovery. It took almost an hour to recover from what was probably 20 minutes of showering.

The other flat top is from the dish washer. I find the 5 needle point spikes interesting they don't line up with usage in the house that I can identify so I believe that is the tank staying warm. I'll be logging for awhile here looking to see if that is the usage. The reason I put working in quotes is that wattage scale on the left is an approximation. I have no way to verify what the tank is actually using and calibrate the sensor. I think my tank is going to use at least 5000 watts, but as I've asked in another thread I don't understand the label on the side.

Anyways I'm very happy with this progress and look forward to determining the watts used on a monthly basis and from that the $$$. The best way would be to calculate the area under the curve but since most of my readings are on or off I'm thinking just counting the minutes where it's on and working KWH from that is easiest.

I also took the time to get the filename updating each night. I went with YYYYMMDD formating as suggested. That took about 2 minutes when I calmed down and though about it.

So far I'm very happy with this project. I've spent $3 in temp sensors $17 in current sensors, $10 in cable and an arduino that you could find for less then $30 if you look hard enough. I now have temp readings from 2 rooms with each additional sensor being $1.50 and 2 lines of code work, A current reading that tracks when my woodstove fan is running, and continuous current readings from my hotwater tank all being logged to a pc from which getting that data online is commonly dealt.

Next step. will probably be getting the data online. opening the CSV file and generating a new graph each time is getting annoying. Plus I want to see realtime readings on my TV...

Not far down the road will be getting the differential temperature controller aspect of the project working. It should be easy in my mind - read the temps if the conditions are appropriate turn a pin connected to the relay on so the relay is activated. If conditions are bad turn it off. This can all be processed in the main loop every 2 seconds like the data collection.

I also need to start seriously thinking about getting a permanent board with proper connectors to pull all of this wiring together. The breadboard is getting ugly to work on with 22 wires coming to it and a crap load of jumpers to share VCC and ground.

strider3700 12-29-10 11:18 AM
I just checked the overnight charts on the hotwater heater and every 3 hours the tank ran for about 3 minutes to maintain it's temperature. So that works out to 24 minutes of operation per day. If it's using 5kw while running then it's using 2 kwh each day. or roughly 15 cents.

<Edit>

I did a bit of thinking on how to calculate total daily usage and came up with a pretty accurate in this case solution. I simply go to my spreadsheet and added a column with =if(wattage>100,1,0) so basically if the wattage being read is above 100 then it's on else it's off. The 100 is used to remove noise from the readings, I get spikes up to 50ish watts which is probably just random noise in the signal. The heater definitely uses more then that when turned on so I'm ignoring them. Anyways this gives me a minute by minute account of if the heater is on or off. From there I just sum the number of minutes it was on and applying the 5kw value calculate the total kwh used during the day.

So yesterday from 12:15 to midnight we used 9.5kwh to heat hotwater which cost roughly 76 cents. This sounds high but our major water usage came in the afternoon so the morning probably wouldn't be much more usage.

This works out to roughly $22.8/month for hotwater or $277/year for hotwater. Assuming I can achieve 70% solar heating which is the commonly thrown around number for my location then my in progress solar hotwater setup will save me $194/year giving me a roughly 5-6 year payback I think. THis is before utility increases and in a house with a baby and a toddler. Long term hotwater usage will go up and the utility has already applied to have a 10% a year increase in costs for the next 3 years.

strider3700 12-30-10 12:00 PM
It was explained to me that I was probably using the wrong value for kw when the heater is running. It's likely that only one of the elements is turned on at any given time so since it's wired for 240V I'm probably using 3800 Watts not 5000. The explanation is good enough that I will recalibrate based on that and redo the math.

Yesterdays numbers are in and the water heater ran for 133 minutes. By far the biggest user was the kids baths and the hotwater load in the washer at the same time. My wife and my quick 4-5 minute each showers made the heater run for about 20 minutes total. THe kids baths and washer had it running for over an hour.

Once again the distinctive every 3 hours temp top up for 3 minutes spike was visible during periods of us not being at home or in bed.

So doing the math based on 133 minutes of 3800 watts means 8.42Kwh costing $0.67 for a monthly total of $20 or a yearly total of $242 This pushes non adjusted rates/usage payback of the solar system out further but it's still worth it.

On the more tech implementation side of the project I'm thinking I need to adjust how I'm handling the readings/data. Averaging on the arduino and dumping the data off every minute rather then every 2 seconds and averaging on the PC should give me the same results but make it easier to store data on the arduino when the PC isn't running. I'm wanting to do a reboot and install some hardware on the PC but don't want to lose data while it's down for 10 minutes.... I'll look into datalogging on the arduino and setting up some sort of system where the PC requests the data from it. That makes a bunch of potential issues go away but adds complexity to get the times synced up.

Daox 12-30-10 12:25 PM
Have you looked into how much onboard memory there is on the arduino and how much time worth of data you can store? I know the newer chips have more memory on them. I'm not sure what one you are using.

strider3700 12-30-10 12:44 PM
It's the at mega with the 1280 chip so 128K of memory for the programs. It's only got 4kb of eeprom that I can read and write to but if I use it efficently thats a bit. My daily csv files come in at 65ish kb without any attempt to minimize their size (It's storing 10 or so decimal points on temperatures....)

I want to lookinto some external memory for it though. I saw some 1 wire memory chips when I was getting samples from dallas I will track them down again and see what they offer or if others have gone a different route.

Hmm I just found that there is a library to use the onboard flash for storage. Since my current programs use 8k of the 128k I can probably get away with using that for a days storage easily enough.

So far I have to say I'm loving this project.

strider3700 12-30-10 12:55 PM
On a totally different note the mail man just dropped off a box of 8 current sensors from Bourns inductive products. I sort of remember requesting these as samples 2 or 3 months ago. They say 7203-RC on them. I'll have to look up the data sheet later. I'm more then willing to try and promote any product that manufacturers send me free samples of.

If I buy the product and it's garbage though Then my online slamming knows no bounds.

strider3700 12-30-10 03:08 PM
I stand corrected. The flash memory on the arduino appears to be not applicable for the storage of data like this. So it's back to an external chip or the 4kb eeprom. Joy.

Daox 12-30-10 03:20 PM
Are you sure? I could have sworn it was... Why can't it be used?

strider3700 12-30-10 11:35 PM
well it appears that you can write to flash however it's only meant to be written to 10,000 or so times. Sounds like a lot but I currently write 1440 times a day so there is a good possibility I'd be pushing it's life expectancy within a month.

It also appears to be a glorious hack of a method to make it work.

apparently I have 8K of ram to work with and since I have an external power plug and a 9V battery I may just check to see how much space in ram I actually have available and store in there.

I guess the first step is to sit down and figure out just how small I can get the data before storage. Some things can be greatly simplified. my current regular row of data looks like this.

2010/12/30 00:55 17.682224 21.12 44.100002 4229.0796

Date time could probably be reduced. next is temp of my office, 17.7 would be fine, 21.1 for recroom temp is also fine and next is 44 watts is what my woodstove is using but it always uses 44 watts or 0 so it could be 1 or 0 same with the 4229 watts for the hotwater could be reduced to on or off. Even if I kept the values for applications where they have changing watts rounding to whole numbers is fine.
so

2010/12/30 00:55 17.7 21.1 1 1 which is roughly half the size already. dumping the date out should also be possible with a little thought or at least only entering it once at the start of each day. I think I've seen somewhere where the arduino has a clock in it so I can track time based on it not based on what the computer says the date is.

Once again this is becoming a question of what do I really need it to do. My PC is never shutdown for long periods. 30 minutes to do hardware work or major OS changes now and again maybe but that's about it. I probably don't need to log days worth of data.

Piwoslaw 12-31-10 12:58 AM
Quote:
Originally Posted by strider3700 (Post 10787)
2010/12/30 00:55 17.7 21.1 1 1 which is roughly half the size already. dumping the date out should also be possible with a little thought or at least only entering it once at the start of each day. I think I've seen somewhere where the arduino has a clock in it so I can track time based on it not based on what the computer says the date is.
You could dump the clock. If you are collecting data every 5 minutes, then the first row after a date change is midnight, the next row is 5 past midnight, the 40th row is 40x5 minutes past midnight = 3:20am, etc. And since you'll already be creating a new file for each day, then the date is already in the file's name. That pretty much takes you down to 2 temperatures + 2 bits of info in each line.

EDIT: Actually, those two bits of information can be encoded in the temperature info. If the first temperature is positive (resp. negative), then the first bit is 0 (resp. 1). Same with the second. Of course, this will only work if the temperatures you are measuring will always be above 0, which is true in this case.

Also, if you know how many bits are used for each temperature reading, and what temperatures are possible, you may be able to encode everything as one number, then use a spreadsheet to decode.

strider3700 01-02-11 01:46 AM
yeah I do horrible things to data to make it fit various needs for a living so there is still a bunch I can do to save space.

On a data note - My wife was sick with what looks like norovirus thursday. Yesterday was hardcore sanitize everything day and the hotwater was flowing. The hot water heater ran for 209 minutes partially due to 4 loads in the washing machine on sanitize. This was 70 minutes more then the previous day. I figured that it was a one time thing but today due to everyone catching up on showers/baths another load of hot laundry and so on we're at 211 minutes with the possibility of one more 3 minute top up before midnight hits. I'm really glad I'm logging and then reviewing the data each day because a few quick measurements here and there just don't come close to telling the whole story.

strider3700 01-12-11 02:27 AM
It's been 14 days of collecting hotwater usage data. A few things I've noticed.

Bath's use a lot of hotwater. You can usually notice when the kids bath days are.

THe heater runs a fair amount to maintain temperature even when set to a very low 108F. Mine runs for 4 minutes every 3 hours roughly. That works out to 1.52kwh of electricity every day just to maintain water temperatures.
in my case that means an electric on demand heater will pay for itself via standby savings in roughly 13 years... I bet a good insulating blanket will be closer to 1 year. I don't have space to fit one on my current tank so I can't test that to know for sure. If you're running your tank hotter then I am then your payback will be quicker of course.

14 days isn't enough time to get a good average use. I've ranged from 211 minutes ran to 103 minutes ran with 1/3 being over 200, 1/3 being near 150 and 1/3 being near below 110. The average is 147 minutes ran.

The major reason I'm logging this is to get an idea of "normal" use and cost so that I know how much I'm saving when I install the solar hotwater setup and monitor the hotwater tank after that. As it currently stands assuming I get the reasonable 70% heating by solar and the electric rate keeps climbing like it did this year(they've requested the same increase yearly for the next 3 at least) then I'll save $200 this year and $1125 over 5 years which should with any luck have me at break even mid year 5. Savings over 25 years is something near $12,000

AC_Hacker 01-17-11 10:12 AM
Strider,

I just ordered a couple of current sensors of the type you are using. I also ordered 20 of the 1-wire temp sensors and also 20 lm335 linear temp sensors and a couple of pressure sensors.

I have in mind to data-log my homemade heat-pump so that I can wring maximum efficiency out of it. The current sensor had been a stumbling block for a long time. I tried to hand-wrap a torroidal core, but realized that it wouldn't work so good for AC, what with Power Factor and all.

Great tips.

Thanks for showing the way.

-AC_Hacker


All times are GMT -5. The time now is 04:30 AM.
Page 1 of 2 1 2
Show 40 post(s) from this thread on one page

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.
Ad Management by RedTyger