Today, the only thing that returns a vote to a user is the status of "Shipped".
Statuses of "Already Exists" and "Will Not Do" should return votes to the user. Additionally, portal visibility should be considered as well. If a user cannot get to an Idea to un-Vote, it should not be counted against the users vote count.
I understand this complicates things when admins can change statuses and visibility settings and doing so might push the users vote count over the limit. I would rather err on the side of my customer not being frustrated.