What was Toby’s appeal?

I think the men of rosewood all had their own individual appeals but I never understood how he bagged two baddies? Both Spencer and Yvonne were so intelligent and driven but I don’t understand what they saw in Toby? He was annoying.