I think needing a certain number of comment contributions to downvote is a good idea, but limiting the upvotes to me is counterproductive.
People like getting upvotes, getting upvotes in itself encourages people to comment. When everyone has a limited number of upvotes per 24h there aren't very many to go around therefore not only is it hard to accumulate the necessary comment contributions to unlock unlimted voting but people will be less likely to comment at all knowing they'll rarely get any votes which is a form of validation. It ends up making the site kinda stagnant, where very little is getting voted on because you run out so quickly.
The interactivity of a site like this is one of its main attractions and in my opinion you should be encouraging people to vote as much as early on as possible so they feel engaged and stay on the site. Having to get 20 comment contributions before you get rid of the restrictions really hampers the user engagement.