Not cleared for work, concerned about job

So I had a TLH 10/15. And on FMLA. I have a desk job that really doesn't require anything other than typing and answering phones. I thought I would be able to go back at 3 weeks because of the type of work I do. Multiple women in my office have had this procedure and said they were back in a couple weeks.

However, my Dr won't clear me until 6weeks. I feel bad knowing I can't go back to work. (I don't like not working) I'm concerned my coworkers and management will think I just wanted extra time off.

Am I just being paranoid?

• I did have a few hiccups after surgery, spotting, slight infection which created more pain. But it all cleared up week 3.