That doesn't stop the "punish" problem. When someone does something to draw the ire of another, they'll not simply downvote the problematic post, they'll downvote several of their posts. Furthermore, if they can tell that said person downvoted another post, they'll upvote it to cancel it out.
Downvoting shouldn't be a thing - period. The human tendency to abuse and game the system out of spite is too great - I've never seen it not abused when both are an option.