James Fitzjames Stephen was a barrister, a legal member of the Viceroy's council in India where he drafted codes of evidence and contract, and later a High Court judge. He wrote Liberty, Equality, Fraternity on the voyage home from Calcutta in 1872, and it remains the most formidable attack on Mill's On Liberty ever written, not least because Stephen had been a utilitarian and knew the machinery from inside.
His central objection is to the harm principle. Mill had held that the only purpose for which power can rightfully be exercised over any member of a civilised community, against his will, is to prevent harm to others. Stephen replies that this distinction cannot be drawn: conduct that appears self-regarding shapes the character of those around it and the moral climate everyone lives in, and no society has ever confined its restraints to direct injury. More fundamentally, he argues that all government rests on force, that law is coercion whatever else it is, and that Mill's liberalism is not the abolition of compulsion but the concealment of it behind a preference for compulsion by opinion rather than by statute.
On the positive side he defends the criminal law as the expression of moral condemnation. Punishment exists partly to gratify the healthy natural sentiment of hatred toward the criminal, and it should — the law gives to that sentiment the same relation as marriage to the sexual appetite. Toleration is appropriate where doubt is genuine, but a society confident of its convictions may properly enforce them.
His account of the other two terms is equally blunt. Equality is a fact about nobody: men differ in capacity and virtue, and arrangements pretending otherwise produce hypocrisy. Fraternity as a universal sentiment is an illusion; real affection is particular, and its counterpart is the exclusion of those outside.
Stephen's India shaped all of this — he thought the Raj demonstrated that good government by a confident minority beat self-government by the incompetent — and the position is authoritarian in a way he never disguised. He is nonetheless the indispensable critic of liberalism from inside its own premises, and every later communitarian and legal moralist, Devlin above all, is working his ground.