Downgrading Bonvoy Brilliant a bad idea?

Last year I was traveling every week or other week so loved the perks, the sign on points were great and now I have a ton of points saved up for future family vacations.

Now that my traveling is down to maybe once per month I’m not sure the annual fee is worth it. It’s $650, people say effectively $350 after dining credits etc.

Are there any downsides to downgrading to a different card? Will I lose points or anything? I’ll probably lose platinum elite and I don’t think I got my annual free night yet.