It's also based on your morality. And the morality of your classmates, coworkers, and cousins. Your best friend and worst enemy and your next door neighbor whose name you can't remember even though you've seen her several times a week for years.
It goes like this:
Morals --> Opinions --> Votes --> Law
On a practical level, we have no "intrinsic" rights. We have the rights that we, as a society, have agreed we should have and that we, as a society, are willing to formalize in legislation and give the force of law.
So how do we determine which rights to recognize as a society? We argue about it. A lot. Sometimes with careful application of already-commonly-accepted philosophical points. Sometimes with snide comments and swearing and screaming. We have strong opinions about what counts as a "right" and who should have which rights and why. And we vote.
Our opinions (and thus our votes) are based on many things, I'm sure. One of those things is our personal moral views.
And why shouldn't that be the case? I would hope that people argue and vote based on what they think is right. Of course I don't always agree with what they thinks is right--I often hugely disagree. And I'll argue with them and vote against them. But I respect that they are acting according to their conscience. Even if I think their conscience is stupid.
Usually when people say "Don't push your morals on me!" they mean "Don't push your religious perspective on me." I can understand saying "Look, we don't all agree that God exists or, if he does, that he is all good, all powerful, and just. Why should the rest of us have to play by the rules of someone we think is imaginary?" Yes. I get that.
But "I don't agree with your religious teachings" is quite different from "Don't push your morals on me!" We all push our morals on each other--that's the only way it works. One person thinks it is immoral for two men to get married. One person thinks it's immoral if two men can't get married. A lot of people think it's immoral to have legal abortion. A lot of people think it would be immoral to outlaw abortion. And on and on. And most of us work to get society to accept, encourage, and even enforce our morality.
In a democracy, laws are based on collective agreements about morality. What else could they be based on?