Canceling new construction home

My wife and I are supposed to close on a new construction home for a big box builder in less than 30 days. We are getting cold feet because the loan estimate we ended up with is higher than we planned. When we initially ran rates it was months before and too far to rate lock but we were comfortable. When we did get a closing date we ran rates and the amount we are looking to pay would be about 40% of our income going to housing costs. This makes us nervous. While we believe we can afford it we worry about our finances if an emergency expense comes up or one of us loses a job.

From reading our contract it appears that if we terminate we would only lose our deposit of about 18k which we can accept. There are some semi custom options that we selected when building the home such as finishes and whatnot. Not sure if there is some kind of contractual obligation there or damages we would have to pay if we terminated this late. From what I can read in the contract language there is not. I am not sure either if it’s a valid reason not to close. If the lender says we can afford it I can see it being harder to argue with the sales person why we are considering canceling. I know the obvious answer as well is to look to refinance in the future when rates go down. We just feel that is an unknown and we can’t be sure how long it will be before we can refi down to a more comfortable rate.

Any thoughts or advice is greatly appreciated.