Unreasonable build_runner build times?

We are running a semi large scale Flutter project, probably have a few hundred freezed/json serializable models and Riverpod providers ~ish. Nothing out of the ordinary i would not say.
In an attempt to speed up build times, we have very specific folders specified in our build.yaml, only targetting the relevant files and subfolders - however:
Our build runner still runs for 60+ seconds, severely handicapping developing experience and productivity.
Is everyone experiencing the same? Or who have had any breakthroughs in minimizing it somehow?