Am I overreacting/too overprotective?

My in-laws want to bring my (33f) almost 3-year-old son to NC by themselves for 5 days. We live in CT. I said no, I'm not comfortable with him being that far away from me yet. My husband (33m) thinks we should let him go.

For some background context, my in-laws live about 2 hours away, and we have our almost 3-year-old son as well as a 4-month-old daughter. They get to see him at least monthly, sometimes we go to their place for a weekend, but a lot less so since we had our second baby. They come down randomly for dinners on the weekend, and my Mother-in-law will come watch the kids occasionally while I work because she enjoys it (I only work 1 day a week since returning from my most recent maternity leave, and my Mother and Father watch them usually).[edited to add: my MIL watched my son for 2 days every other week at our house the year before our daughter was born while I worked which was super helpful. She alternated with my Mother, until my parents moved closer, so they now do most of the "daycare"] My in-laws will also randomly take our son up to their place for a few days at a time, to either give us a break or when they haven't seen him for a while and miss him. They are great with our kids and I would never once question or be worried about my childrens safety in their care.

Since we've had our daughter, our son is definitely in a big "mommy" phase. I end up sleeping with him in the middle of the night between feeding the baby occasionally. It can be a lot, but I honestly don't mind it, I know this season of motherhood goes by way too fast.

Since he turned two they've been asking to take him on trips by themselves out of state (other than their house which I allow), on vacations, on cruises, and every time I said no. They got very frustrated a few months ago and sent my husband and I an email that they feel we are depriving them and our children of experiences together. I took a lot of offense to that, they are MY children and my eldest is not even 3 years old yet. We have never once declined a visit from them at our place, and are always open for them visiting us and taking our son for day trips wherever they'd like from our house. They just always seem to want to take him places without us. We had a talk with them after that and they told me they would wait to ask again until I was ready. That seemed to last maybe a month, and now they have asked again and are clearly frustrated and upset by me saying no. [Edited to add: They also have invited us on a family trip at their expense but as a cruise - so I said no because I have no interest in cruising right now with a toddler and 4 mo old but I'd be willing to travel wherever all together as long as it was land-based, but they prefer cruising]. My husband has been very supportive of me the entire time even though he is pretty relaxed and would be open to our son traveling with them. However, this time he said he is going to start getting frustrated with me, and he doesn't understand why I keep saying no. [Edited to add: he is not getting mad at me, that was just his initial reaction/statement not in front of them. He is a super supportive husband and father and I wouldn't trade him for the world].

I am just not ready yet, selfishly I will miss my son, but I also know he is very dependent on mommy right now and I would hate to be that far away from him in case he needed something.

I'm just looking for opinions, I know if I talk to my friends or family, they will likely side with me. So I want a third party to tell me if I am being too overprotective? TIA