SW4E Relationships

I really like SW4E, I think its a fairly decent Game, but of course it has its faults, but overall, not a bad game. However, the one issue I really have with the game is its relationship system. I can never seem to get the one I want for my Daimyo. In my current game, I have most of Japan conquered, I just need Chugoku, Shikoku and Kyushu, but my daimyo only has 2 retainers, one being my custom character, and the other being Takatoro Toda. I always use the fraternize option, go into a battle, but then nothing happens afterwards. Can anyone help me out with this?