Hot air usually moves to where cold air is. It's the dynamic of pressure which causes convection. Stack effect creates pressure but it doesn't always mean that the air will move up. The cooler air moves in through soffit vents, then gets heated by the roof and pressure drives it both into the house and out the ridge(or turtle/etc vent).
Since you seem to want the use of the attic fan with periods between that time to not have the airflow, my suggestion would be to buy some thick plastic and either use a reusable double sided tape or glue a magnet to the perimeter of the plastic and put it in place when not in use and remove it prior to turning on the fan. Just be sure that when you are not in the whole house fan season that you cover the entire unit with an insulated box that is sealed with a gasket so it isn't an energy sieve.
The easiest solution though to making sure you are always sealed and insulated when the fan is off is to have one of these instead.
Indoor air quality, whole house fans, exhaust and ventilatio
In my house, I ended up buying 4 box fans and I stack two on top of eachother in my slider windows. Two as an intake for two different windows downstairs with the rest of the window open and it really moves a pile of air. Saved a ton of money over the whole house fan and takes two minutes to open windows and place the fans in and power them up and as soon as I'm done the house is as sealed as it always is.