cabin on the coast and Doyle's wife comment

At one point Daniel the faerie king says that twenty years of service is short enough, "like Doyle's wife" and the only Doyle I can think of is Arthur Conan Doyle, who had a weird little love triangle with his ailing and dying first wife and his second wife, who actually worked as a medium - they got involved in spiritualistic stuff- at one point Doyle even championed some rumors of what was ultimately a fairy hoax. However, I can't see any descriptions that list one of his wives as short. What does Daniel mean here? Or is it a different Doyle, with a short tempered wife?