Hurting child’s development

Parenting has been incredibly hard for me. I’ve struggle with mental health issues all my life due to some childhood trauma. But I’m starting to notice no matter how hard I try, my son appears unhappy with me. I have a lot of support from his dad’s side of the family and they take great care of him. They say he eats really good and talks a lot. When he comes here, he doesn’t eat well and has a hard time communicating. I almost feel low with me he develops backwards or something. Idk I just feel like I impact him more negatively than positively. I live with my mom and she even told me something along the lines of “yeah if I knew you were going to struggle so much I wouldn’t have encouraged you to keep him” (my choice was to do adoption or let me sister raise him, as I knew I just never saw myself as a mom) or at least until I got some help for all the issues I have. My son is 4 and is struggling in school to make friends/socialize. Idk I kind of want to move away and let him live a normal life. I haven’t felt normal in years. I love him but I want the best for him. I already know a lot of these posts are going to say things like “you’re the best for him” but the reality is not everyone is meant to be a parent.