It should cost people to downvote, and to do this you should lose one karma point for every downvote. This is how Gab operated when they allowed down voting, but they ultimately eliminated down voting which I disagreed with, but whatever. Over on Poal, all votes are logged so you can see who is downvoting you.