Not sure of the rules but my home has the boundary fence built on top of the retaining wall. If I understand you right.
With your point 3, it may not de-value your property but would make it more difficult to sell, if you ever sell, as most buyers will be put off by this.