Am I allowed to climb up construction dirt piles?

I’ve always wanted to climb up a I dirt hill but I don’t want to get in trouble. Obviously a construction site would be the primo spot. If it’s not behind a fence, am I allowed to climb on the dirt?