Outsulating may be in the future, but not the near future. Especially not if this works well.
I'll definitely have to weigh the differences between spray foaming the cavity with a DIY kit and doing foam board. I think both are good options for sealing though.
Here is a bit more info on the office. I will get pics up yet! But, here is the layout of my house. As you can see, only one wall of the office is an exterior wall. The room isn't tiny at about 14.5' x 14'. But, this should be a relatively easy/small project to test out how I want to do all the walls upstairs.
The angled upstairs ceilings will definitely need some good insulation, much better than what is there now. I haven't even really thought about how I'd do those yet.