The Civil Libertarian tradition is the oldest line in the liberal inheritance and the one your politics still patrols: the border the state may not cross. Where the rest of the libertarian family argues mostly about the economy, your concern is older and more intimate — the sovereignty of the person over conscience, belief, speech, and the conduct of a private life. You hold that there is a zone around each individual into which no government, and no majority however large, has the authority to reach; that the right to think, say, worship, and live as one sees fit is not a privilege the state extends but a limit on what the state may do. And you hold it with a forward-leaning temperament: a society that keeps widening that zone, that lets people live authentically and sheds inherited constraints, is a society getting better.
The foundation is John Locke, who drew the line first and drew it around conscience. Against an age that took for granted that the sovereign should enforce the true religion, Locke argued in A Letter Concerning Toleration that the magistrate has no authority over the soul at all — that belief cannot be coerced, that the care of one's own conviction belongs to each person and to no ruler, and that rights to life, liberty, and conscience exist prior to government and set the terms it must respect. The radical move was not any particular freedom but the structure: rights come first, power comes after and bounded.
Voltaire took the principle from the study into the street and made it a campaign. Across the Philosophical Letters and a lifetime of pamphlets, he fought religious persecution, judicial torture, and the censorship of crown and church, championing free expression and tolerance not as abstractions but as the difference between a civilization and a machine for crushing dissenters. He is the source of your instinct that orthodoxy backed by power is the permanent enemy, and that the individual's right to speak heresy is the test of whether a society is free.
James Madison did the thing the others could only argue for: he wrote the line into law, and he located the danger more precisely than anyone before him. In the Federalist Papers and then in the Bill of Rights, Madison saw that the threat to liberty in a republic was not only the king but the majority — that a popular government could oppress conscience and silence speech as thoroughly as any tyrant, and more legitimately. So he built protections a majority could not vote away: religious liberty, freedom of speech and press, rights held against the public as much as against the state. Much of what you now take as given exists because Madison reasoned that some freedoms must be placed beyond the reach of a show of hands.
John Stuart Mill gave the tradition its philosophical capstone and its rule of thumb. On Liberty named the principle your instinct already obeys: that the only legitimate ground for restraining an adult's freedom is to prevent harm to others — that society may not coerce a person for his own good, in belief, taste, or private conduct. But Mill's deeper argument was that liberty is not merely a shield; it is the engine of human progress and the discovery of truth. Silence an opinion and you may be silencing the truth, or robbing even a true belief of the vital understanding that comes from defending it. Free individuality is how a civilization keeps learning. Wilhelm von Humboldt, whose The Limits of State Action Mill placed at the head of On Liberty, supplied the ideal beneath it all: that the end of a human being is the fullest and most varied development of his individuality, and that the state's whole business is to leave the room in which that can happen.
What you inherit, then, is a politics built around the person rather than the market — which is why you can be flexible about economics and absolute about speech and the private sphere. From Locke, rights that precede and bind the government. From Voltaire, the campaign against persecution and orthodoxy. From Madison, the discovery that the majority itself must be constrained, and the law that constrains it. From Mill, the harm principle and the argument that liberty is the condition of progress and truth. From Humboldt, the vision of a life of fully developed individuality that all of it serves. The throughline is the conviction that there is a self-governing core to every person that no authority may colonize, and that a free society is measured by how much room it leaves for people to think, speak, and live unlike one another. The tension you carry is the one Mill bequeathed and no generation has settled: exactly where harm to others begins, and what to do when one person's authentic life seems to crowd another's — the boundary that must be redrawn in every era because the cases keep changing. But the principle does not move. You are the strand of liberty that has always understood freedom as something owed first to the conscience, and only then to the wallet.