It really is like putting a band-aid over a bullet hole lol.
I did pull off one of the trim panels to take a look at what there is behind it. This is what was there.
I think I'll just be bumping up the priority of gutting the upstairs to get insulation into the walls. At that point, I'll take care of the windows and the walls.