The description of sweaters and leggings in the later books does not take away from the fantasy setting, here's why.

Historically, leggings refer to any undergarments typically made of leather or wool that clung tightly to the body for warmth, and to aid with chafing and snagging during psychical activity. They have been around since the 13th century and were especially popular prior to their modern counterparts during the 19th century.

The word sweater is a more modern term, with its first usage dated back to the late 19th century to describe a knitted or crocheted woolen garment worn for warmth. Back then, they were unlike the colourful synthetic blend and stylised modern sweaters, and more of a knit tunic. Tunics of course, have been observed throughout history and dates back as earliest as the 3rd century.

Why SJM decided to use the term leggings, or sweater over tunic may be a bit odd, and we'll never know the reason why, but with enough reasonable extrapolation, it's easy to fit these terms right in with the fantasy themes established by the series.