OK so it appears to be another bug in the great and powerful 9.6!!
This morning I upgraded to 9.6 SP1 and so far all the agents I have updated as well to SP1 now show correct details and if I check "C:\Program Files\LANDesk\LDClient\CurrentDownloads.log" if an agent is needing to use a Preferred Server it is showing that it is downloading files from the correct one.
SP1 also fixed a few other issues with agents that I was having so hopefully life will get a little easier.