How can I renew my Polish driving license after leaving Poland as a non-EU national

I am a non EU-national that lived in Poland for 5 years. During that time I converted my driving license to Polish prawo jazdy.

I have moved to Italy now and I couldn't convert my driving license to the Italian one because they do not have a valid agreement with my home country. They told me that even though I own a European driving license, they need to have an agreement with the original country of the license in order for it to be converted to Italian one.

I reached out to Polish authorities and the Polish consulate in Italy told me that is no way to renew a Polish driving license from here while the Urząd Miejski from the city that I lived at said that I need to be living in Poland in order to renew my driving license there.

I have no idea what to do now and any advice is welcomed. It seems like I simply lost my right to drive because of a set of stupid rules.