RDS 2012 Connection Broker and round robin DNS?

Hi,

I have a set-up with three session host servers and one of them is also a connection broker. Round robin DNS is configured for the three servers and the clients, who are all on our internal networks, can connect just fine to the farm name. My questions is, is there any point in having round robin configured since the connections still need to go through the CB? Could i just have one DNS-record for the farm name pointing to the CB? My initial though was that round robin would add redundancy, but does it realy? I've tried to find answers on this but none of the articles and posts i've found are realy clear to me.
Quite often the initial connection for clients is slow and my theory is that it's caused by the RR DNS records.