I was at a screening of this the other night that had a Q&A with the (Irish) producer afterwards. I was hoping someone would ask her about that scene but no-one did. I regret not doing so myself.
One thing that was interesting though was that she said the casting of black actors in the roles of the groundsman and the maid was entirely accidental i.e. there were not specifically looking for POC for those roles - they just happened to be the best people for the parts - and those parts of 19th Century England were much more racially mixed than people assume anyway. I found this surprising as I thought the whole film was rooted in some sort of allegory of slavery and racism and so assumed it was very deliberate (the Bristolian accent on the boy for example).
I really liked it ... but do understand your objections.