Here is a GooglMaps 3D view looking South;
The big tree is over the top of two apple trees and shading the plum enough that it never bears fruit (We have the same kind of plum tree out back that is usually overflowing with some of the best plums I've ever tasted).
So it looks like having it removed altogether might be the best answer here. $$$