Hi richuu. Welcome to the boards.
A few things I can suggest, one of which may not be possible withough spending a lot of money to have someone do it, or spending a lot of time cutting into walls and then repatching after you're done. I'm suspecting it's as simple as you're ducts losing air. I watched an episode of Holmes on Homes and the 2 guys had the same problem that you seem to present. All he did was cracked open some walls where the ducts where and then had someone come in and do the smoke test where they blow smoke into your vents to see where the leaks were...well they had a TON. And they even had one hole in their main vent where an offshoot use to be and was never covered up. So they just taped all the vents up, covered up and taped everything they needed to and turned the air back on...and voila. Nothing magical, just too many leaks. So if in the winter it doesn't get warm up there, and in the summer it doesn't get cool...then I'd suspect that your ducts just need some sealing up.
And the other thing, as you mentioned already, is to go into the attic and see what kind of insulation you have up there. And if need be, add some more.
That's all I can think of for now.
