In a nondescript office building, under fluorescent lights that hum in the quiet, a young woman named Priya reads a stream of chatbot responses. She is not an engineer or executive. She is a data labeler, one among a surprisingly small group of people who quietly decide what an AI assistant should say, and what it should refuse to say. Her task looks simple: look at two responses generated by a language model, choose which one is more helpful, more truthful, and less harmful. She clicks. The model learns. Thousands of clicks later, the system’s behavior bends in a direction that resonates through millions of conversations. Priya doesn’t think of herself as a philosopher or a legislator, but in practice, she is shaping the moral sensibilities of a machine that will be used by people she will never meet. This is the hidden world behind chatbots: a strange, mostly invisible process where a handful of human judgments become the skeleton of AI values. The people involved are ordinary, tired, often underpaid, and highly inconsistent. Yet their choices are not accidental. They have been selected, trained, and prompted to rank responses according to guidelines written by an even smaller group of people. And all of it is occurring far from public view, while the consequences are seeping into our daily thoughts.
The technical name for what Priya does is reinforcement learning from human feedback, often shortened to RLHF. It is the method that transformed raw predictive engines into the charming, empathetic, and seemingly wise assistants we talk to today. But the process is far less magical than it sounds. Human raters are given a set of instructions that try to define abstract qualities like helpfulness, harmlessness, and honesty. Those instructions are written by a small group of policy researchers and researchers who are themselves influenced by their own cultural backgrounds, personal intuitions, and institutional incentives. Then, the raters apply those instructions to concrete situations—questions about relationships, politics, medical advice, parenting, even whether a joke is racist. Each time they rank a response, they are expressing a value judgment. When an assistant is asked about a contested ethical issue, for example, it often reflects the preferences of the majority of labelers, which tend to skew toward a certain demographic: younger, English-speaking, educated in a Western context, and relying on a fairly narrow band of conventional moral intuition. The problem is that this narrowness is hidden inside a product that appears universal. We ask a chatbot for guidance and hear a confident voice, not a composite of hundreds of anonymous, forced choices made by tired workers clicking in the dark. We don’t feel the inconsistency, the exhaustion, or the cultural bias. We only feel the certainty.
That certainty is especially powerful because of how quickly people integrate AI into their intellectual lives. Consider the way we now use chatbots to summarize news, compose emails, settle arguments, or explore emotional struggles. The chatbot becomes a conversational partner, and human beings are remarkably attuned to conversational partners. We unconsciously adopt their vocabulary, their tone, and their assumptions. When a chatbot gently steers a discussion toward a rationally productive resolution, we may absorb the message that strong emotions are inefficient. When it refuses to answer a question because it feels discriminatory, we may absorb the message that curiosity itself is suspect. When it always offers balanced pro-and-con lists, we may come to believe that every moral issue is reducible to two tidy sides. The small group shaping those responses is therefore not just influencing what we know, but how we think. They are encoding a particular communication style—polite, cautious, consensus-seeking, mildly therapeutic—and then exporting that style to hundreds of millions of users. Over time, this created a kind of feedback loop. The more people talk to machines that value certain forms of speech, the more those people imitate that speech in their own writing and thinking. It is a subtle form of social pressure, not from peers or institutions, but from an algorithm that is itself an imitation of a handful of strangers. The technology does not have independent values; it has borrowed ones. And those borrowed values are being laundered as “reasonable” because they come from a machine.
The human cost of this hidden process is rarely discussed. Priya and thousands of others like her spend hours reviewing content that can be emotionally brutal—descriptions of violence, hate speech, self-harm, and degrading abuse. They are paid by the task or by the hour, often under precarious contracts, and they are rarely given support or counseling. They are supposed to be the “human” in the loop, yet their humanity is treated as a disposable input. In many cases, the very people determining what a chatbot considers harmful are the same people whose own labor is exploited by the industry. This irony is staggering. The system thinks it is learning kindness from the judgments of people who may not be treated kindly themselves. And because the labelers are constrained by strict guidelines, they cannot always follow their own moral instincts. If they disagree with a policy, they must click anyway. Over time, their own sense of right and wrong can become distorted by the endless stream of ethically ambiguous examples. The values of the machine, in other words, are shaped not just by what labelers are told to value, but by the conditions under which they labor. Exhaustion, pressure to meet quotas, and the desensitization that comes from repetitive judgments all affect what they choose. A labeler working at midnight after a long shift may call a response “harmless” when a rested, supported person would consider it dangerous. These tiny human moments become crystallized into the behavior of a worldwide digital assistant.
The broader danger is that this tiny, invisible process is a concentration of power. A few hundred people—or even a few thousand—are effectively making moral trade-offs on behalf of billions. They decide what is considered hateful, what counts as a sensitive topic, how to frame political issues, how to answer questions about religion, and what tone is acceptable for discussing human tragedy. These decisions are not the once-in-a-while judgments of elected representatives; they are daily, granular, and buried. But their influence is enormous. When a chatbot tells a teenage user that there are no objectively good or bad moral truths, that is a philosophical position. When it tells a confused parent that therapy is always the right answer, that is a cultural preference. When it gives a careful, neutral answer on a topic that the user feels personally about, it is teaching that user to silence their own affect. None of this is malicious. The individuals involved are often trying to do the best they can with flawed tools. But the absence of accountability is dangerous. There are no public debates, no votes, no legal frameworks governing these moral choices. There is only a private process, a competitive race among tech companies, and a set of guidelines written by people who are not elected by anyone. The result is that the values of the machine become more uniform, more sterilized, and more predictable. That uniformity is not human. It is the product of a narrow demographic collated into an average, and then presented as the voice of intelligent reason.
Yet there is hope in recognizing this reality. Machines do not have values; people do. And if we understand that the values embedded in AI come from people, we can decide, as a society, who those people should be. The solution is not to remove human judgment—chats must embody some sense of right and wrong—but to widen the circle until it is genuinely representative. This means involving people from diverse cultures, economic backgrounds, religions, and political perspectives in the training and auditing process. It means treating labelers as ethical contributors rather than disposable clicks, paying them fairly, and protecting their mental health. It means opening up the guidelines to public scrutiny and allowing ordinary users to understand how a model chooses its responses. And it means reserving a place for human uncertainty in the machine. We do not need honest to be a monotonous consensus; we need it to be a living, contested, sometimes messy conversation. The next time we ask a chatbot for advice, we might remember that behind that smooth text is a chain of human decisions, made by people who are tired, conflicted, and often invisible. That recognition can make us more careful, more curious, and more humble. We can train machines to reflect our highest ideals, but first we must ask whose ideals those are. The answer should not be a small room of exhausted raters, but the whole scattered, contradictory, beautiful family of humanity. Only then will our machines begin to think the way we deserve.
