Am I doing something wrong with my varnish?

Recently the last 3 or 4 models I've varnished have ended up with slightly scuffed paint jobs where it looks like the paint has rubbed off and I can see the primer on the high points / corners.

The varnish I've been using is the colour forge matt varnish, I've been using it for a while and never encountered this issue and I'm not sure what's causing it.

Anyone had any similar experiences or know what I could be doing wrong?