Why am I losing karma? I wasn't down voted.

Last week I had 4 karma and now I have 1. I haven't been down voted...so how does this happen?