The Yorkes actually had more portraits painted of their staff than of their own family, and also wrote lengthy inscriptions for them in verse, which led to them being regarded by their contempories as whimsical to say the least.
At the time it was the norm for a country house owner to have expensive portraits painted of his horses or prize winning bulls – they were, after all, the product of careful breeding like their owners - to extend the treatment to generations of servants led to the Yorkes being considered an extremely eccentric lot.