The solution is simple, don't pump all the heat/cool out, but still bring in fresh air.
This is what a
heat recovery ventilator (HRV) is for. Its a heat exchanger that transfers the heat/cool from the outgoing air into the incoming air so that you don't loose as much heat/cool. Yeah, they aren't 100% efficient, so you still loose some heat/cool, but not nearly as much.
We have a good DIY thread about them here:
http://ecorenovator.org/forum/conser...exchanger.html