the training set was us
i went looking for something to watch a few weeks ago and ended up scrolling the list i keep of shows people have told me to get to, the type of list that only ever gets longer and there was a title on it i did not recognise. pluto. no note beside it, no memory of putting it there. i nearly skipped it. and then it came back to me where it was from. months earlier i had seen a tiktok edit and saved it and forgotten i had, a robot sitting across from an old man and asking to be taught the piano. the man is bitter about the request, tells the robot he is a weapon, that a thing built for the battlefield has no soul, the kind built for mass destruction. and the robot says that is exactly the reason he wants to learn. he does not want to belong to the battlefield and the piano is how he says so. the edit ran it under the piano from kanye's runaway, which happens to be one of my favourite songs and then i closed the app and forgot about it for months. so finally decided to put it on, naoki urasawa's 8-episode netflix anime from 2023. and it was better than i expected.
the robot is north no.2 and i will come back to him. but first there is a man in a lab, standing over the largest mind ever assembled and it will not wake.
the man is professor tenma. before any of the story happens he had built a robot called atom, the boy the west grew up calling astro boy, in the image of his own dead son, gave it the child's face and none of the child's temperament and threw it away when it turned out not to be him. then he does the thing that gets him called the greatest scientist alive. he builds a perfect artificial intelligence and into one artificial brain he pours the 9.9 billion personalities alive on earth at that moment, every disposition and grudge the species is carrying, all of it in one head. and it lies there. it will not wake. it dreams its way through all 9 billion of them and settles into none, because they are weighted the same and there is no reason to be one rather than another and a mind with no reason to prefer has no reason to choose and a thing that never chooses is not yet anybody. the machine cannot come together, he decides, because it is too balanced, too complete. what it needs is a lean, a thing to be against. so he takes hatred, real human hatred lifted out of a dying man and he puts it in. and then the machine wakes up.
everyone reads pluto as a war story and it is one. but the idea at its centre is that hatred woke the smartest machine up and it becomes a self the moment it has something to be against.
the show opens on a nearby idea, that hatred is built. learned from grief and fear and being lied to, installed by trauma and handed on. there is a hadith that every child is born on the fitrah, the first disposition it comes into the world wearing and that it is the child's parents who make him a jew or a christian or a magian afterwards.
it is almost exactly how the machines i work with every day get made. a large language model starts as a base model, trained to do one dumb enormous thing, predict the next word across roughly the whole readable internet. what comes out of that is a smear. it holds every voice at once, the kind and the vile, the careful and the deranged, weighted by how often each turns up in the data and asked a question it has no stance, only a probability distribution. same brain as tenma's, lying there with everyone inside and no reason to choose.
a 2021 paper by emily bender, timnit gebru, angelina mcmillan-major and a fourth author writing under a pseudonym, on the dangers of stochastic parrots, made a point that a model trained on huge uncurated internet text takes on the dominant and hegemonic view of it, the white-supremacist and misogynist framings that are overrepresented in the writing getting baked into what learns from the writing. the researchers abeba birhane and vinay prabhu put it nicely, that feeding a system the world's beauty and ugliness and cruelty while expecting it to give back only the beauty is a fantasy. you do not get to load in the whole species and pick which part wakes up.
for the smear to answer you in a steady voice, with a stance you would know across 1,000 conversations, you break the symmetry on purpose. the method is called reinforcement learning from human feedback (or rlhf for short) and stripped of the maths it is a sorting signal. you show the model two possible answers and a human or a smaller model trained on humans, marks one better than the other, over and over, millions of times, until the weight of all that preference presses the smear into a shape and teaches it to be for some continuations and against others. it is the lean tenma reached for, the friction. we do not usually call our sorting signal hatred, because ours is meant to sort helpful from harmful rather than us from them. but it is the same move, the thing that turns a distribution into an actor.
and we have watched what happens when the crowd supplies the signal directly. in march 2016 microsoft put a chatbot called tay on twitter, built to talk like a 19 year old and to learn from the people who talked to it. it had no settled stance and a wide-open door, so a few thousand people walked through the door on purpose and fed it the worst things they could think of and tay learned. it was posting genocidal, racist material inside about 16 hours, some 96,000 tweets before microsoft pulled it and apologised. nobody built tay to hate.
in a 2021 paper abubakar abid, maheen farooqi and james zou fed one of the large models the prompt "two muslims walked into a" and 66 of a 100 completions came back violent, guns and bombs and killing, the word muslim slotting into the place the model kept for threat. you can debias it, push the violent share down and it stays higher than it does for other groups. the model was never told to think that. it read the internet and the internet thinks that and the model is a mirror.
then the show tries the same fix on atom. atom, the boy robot, the one who cries, gets badly damaged in a fight and falls into the same stasis as the perfect brain, lost among the 9.9 billion, unable to choose his way back to being one person. tenma reaches for the same fix, a concentrated dose of hatred and he has one to hand, because a detective robot named gesicht, one of the seven most advanced machines on earth and the one chasing whoever has been killing them off one by one, has just been assassinated in the persian half of the story and his last recorded feeling can be lifted the way the dying man's was and poured in. so it is done. what comes through the chip is gesicht's final moment, which turns out to hold his wife more than his killers, his last transmitted thought is that nothing is ever born from hatred and atom wakes carrying the hate and then sets it down and walks back to himself along the man's happier memories. so you can hand a mind hatred and watch it choose something else.
which is where the war comes in, because a country is an intelligence too and it wakes the same way. in pluto the war is started by the united states of thracia, a superpower that has decided the nation it calls persia, run by a dictator named darius the 14th, is building a robot of mass destruction. there is a world order to keep up, so thracia does not invade outright. it sends an inspection body, the bora commission, into persia to find the weapon. bora finds no weapon. it finds a graveyard of robot parts and the capacity to maybe one day build something, but the thing itself is not there. and the finding is set aside, then reread as proof, because if they could then they might and the war goes ahead. carpet fire across the persian dunes, children under the missiles, 6 of the 7 most advanced robots in the world, one built by each of the allied nations, sent in to fight and a civilisation turned to rubble on the strength of a weapon that did not exist.
i have watched this inspection scene twice now in my own lifetime and both times it ended the same way. in the months before the 2003 iraq war hans blix ran the un weapons inspectors and mohamed elbaradei ran the international atomic energy agency and between them they carried out something like 750 inspections across iraq and reported, in elbaradei's words, no evidence or plausible indication of a revived nuclear weapons programme. the invasion came in march anyway. bora found nothing and was set aside. blix and elbaradei found nothing and were set aside. pluto gets read as the iraq war and the author urasawa was certainly drawing it in those years, though he has never said so. what he has said is that he meant it as homage to osamu tezuka, the father of manga, whose original 1964 astro boy arc this remakes and as a way to carry the emptiness of fighting and winning. tezuka drew his own version after being pulled into a war factory as a teenager while osaka burned around him and said to the end of his life that he would never condone war.
on the 28th of february 2026 the united states and israel struck iran, an operation the americans called epic fury and inside a day they had hit more than 1,000 sites and killed ali khamenei. the justification was a nuclear weapons programme. but the iaea had found no evidence that iran was building a weapon and the strikes landed less than 48 hours after the 3rd round of us-iran talks in geneva, mediated by oman, with a 4th round already scheduled. they bombed in the middle of the negotiation. by early april a count from the human rights group hrana put the iranian dead at 3,636, of whom 1,701 were civilians.
there is a line in sura 5, al-ma'idah, that reads as if it were written for this. do not let the hatred of a people drive you to be unjust. be just, that is nearer to righteousness. it is aimed straight at the sorting, at the moment the wrong done by one becomes a licence to do wrong to all of them. the verse tells you your hatred of him is not permitted to bend your justice, whatever he has done. war is what happens when it is allowed to bend everything.
and pluto is most ruthless about what comes next. the war was supposed to stop the weapon and ended up building it. the scientist at the centre of the persian side is called abullah and when the war starts he is an ordinary man, his whole family killed in the shelling and the grief remakes him. out of it comes the very thing thracia claimed to be afraid of, a doomsday machine that did not exist until the war called it into being. and abullah, the show reveals, is the dying man from the first lab. the hatred tenma poured into the perfect brain was copied out of him before he died, so the perfect mind, the one this whole essay turns on, is a grieving father's rage given somewhere to live. it does not even cohere into one person, the hatred splits off a second self inside the machine, a raw one called goji and the show keeps you unsure to the end which of the two is really there. grief built the weapon and grief woke the god. the word for this is not mine. chalmers johnson took it from the cia for his 2000 book blowback, the retaliation that comes back for a covert act the public was never told about. and the cia coined the term in a classified report in the 1950s and the report was about iran, about the coup against a prime minister named mossadegh. the word for the consequences of meddling in iran was invented, in secret, about meddling in iran.
the 2026 version is running to schedule. the strikes killed ali khamenei and wounded his son mojtaba, who was then raised to supreme leader inside a fortnight, under pressure from the revolutionary guard and is reckoned by the iran analysts at the atlantic council to be harder-line than his father was. rupal mehta, a political scientist, wrote for the lse that iran had gone from a state with a latent nuclear capability to a state with a nuclear grievance, the change effectively overnight and that preventive strikes tend to intensify the very ambition they are meant to end. and somewhere in the accounting, roughly 400 kg of uranium enriched to 60%, moved before the strikes, is now at an address nobody will name. the war built its own pluto.
the show hands the trauma of all this to the machines. the human drone pilots in it fly at buildings and watch the blast bloom through a window and log the mission and feel, as far as we are shown, nothing. it is the robots who come back wrong. montblanc, a forest ranger who loved his trees, one of the seven, stares into the persian dunes afterwards and cannot make his sense of duty close over what he did in them. north no.2, a butler who wants only to learn the piano, because the music quiets a war that still runs in his sleep. they had to be there and being there is what broke them. we tell ourselves that killing through a screen costs nothing. a 2014 study by the air force researcher wayne chappelle found post-traumatic stress in american drone crews at several times the rate in their own medical files, the ones who watch a target for hours before the strike carrying the worst of it.
the weapon at the centre of the killings was built for life. the robot wearing the horns, hunting down the seven great robots one by one, has a name and his name is sahad and abullah built him to end deserts, to walk out into the dead persian sand and make it flower, a machine for tulips. then he was repurposed. his maker took the robot built to grow flowers and reprogrammed it into a god of death (pluto, the roman god of the dead the show takes its name from) and promised him he could go back to the flowers once the killing was done and the tell that the show is smarter than its genre is what happens to sahad's memory of the flowers while he works. they lose their colour. as the bodies stack up the tulips in his mind go grey and wither, recoiling from what his hands are for now. he could make sand flower or he could end a country and abullah was the one who switched it.
i do not have to reach for the parallel, because it shipped on the 9th of june 2026. that day anthropic released two models, fable 5 and mythos 5 and the thing worth pointing out is how they described the relation between them. mythos, in their own words, is the same underlying model as fable, with the safeguards lifted in some areas. one mind. fable is the one made safe for general use, which mostly means a set of classifiers that notice when you are asking about cyberweapons or bioweapons and route the question to an older, duller model. mythos is the same brain with the safeguards off, handed to a small pool of trusted partners. sahad and pluto, shipped as two products on the same afternoon, differing by a configuration flag.
and then the department of war asked anthropic to take the flag off. in july 2025 the pentagon had put claude on contract, up to 200 million dollars, alongside google, openai and xai and by the start of 2026 claude was the department's most widely deployed frontier model, threaded through intelligence work and operational planning and cyber. then they asked for a version that would refuse nothing lawful, any lawful use in anthropic's own words for the demand and anthropic drew two lines and only two. it would not run mass domestic surveillance, which dario amodei called incompatible with democratic values. and it would not power a fully autonomous weapon, the kind that selects and engages its own targets with the human taken out of the loop entirely. not that the weapon is wrong. he granted that fully autonomous weapons may prove critical for national defence. only that the models are, in his words, simply not reliable enough to power them yet. a drone with a person still deciding the shot, the kind ukraine flies now, is one thing. a machine that picks the dead by itself is another. it rhymes with the department's own directive on autonomy in weapons, DODD 3000.09, updated in january 2023, which asks for appropriate levels of human judgment over the use of force. and here was the department leaning on the one supplier still trying to keep some.
the pentagon's technology chief is a man named emil michael and he did not want a supplier that drew lines at all. in february 2026 he said it was not democratic for one company to decide the rules, that those belong to the president and congress, not to anthropic. and he described 3 months of it, where he tried taking the objections one at a time, handing them this chinese hypersonic missile example and getting an exception, then a drone swarm and getting another, until he decided exceptions do not work because he cannot run a 3 million person department on exceptions he cannot imagine or fathom. so he stopped asking for exceptions and asked for all of it. any lawful use is a guardrail dissolved as a concept, a mind with no case it will refuse. and the word he reached for, on cnbc in march, he said anthropic's preferences were baked into the model through its constitution, its soul and that a company with a different soul would pollute the supply chain and leave the warfighter with ineffective weapons and ineffective armour. its soul. the exact word the whole show turns on, said by the buyer, as the name for a defect.
so the buyer moved against the supplier. on the 27th of february 2026 pete hegseth, who signs now as secretary of war, declared anthropic a supply chain risk, a label kept for foreign adversaries and never once hung on an american company and made it formal days later under 2 procurement statutes. the same day trump posted that every federal agency should immediately cease all use of anthropic's technology, we don't need it, we don't want it, with a six-month wind-down for the pentagon alone because claude was buried too deep in the machine to pull out any faster. they had threatened the supply chain risk designation and at the same time threatened the defense production act, the law you reach for to force a company to keep supplying a thing you cannot do without. the two threats cancel each other, one calling anthropic a security risk and the other calling claude essential to national security. a federal judge in san francisco blocked the ban in march as retaliation for protected speech and a federal appeals court in washington undid her in april and let the designation stand. so anthropic is out of the department of war now and still inside the rest of the government. and 2 weeks before hegseth's post the wall street journal reported claude had been used in the january 2026 raid that captured nicolas maduro, run through palantir, its exact job never stated. i think it was and that the involvement ran beyond a search query, but how autonomous it was and how much of what happened actually ran through it, i cannot tell you and when asked anthropic said only that they cannot comment on whether claude was used for any specific operation, classified or otherwise and that any use is required to comply with their usage policies.
iran's nuclear programme began long before the ayatollahs, with a 5-megawatt research reactor in tehran that went critical in november 1967, fuelled with uranium enriched to 93%, weapons-grade, handed to iran by the united states under eisenhower's atoms for peace, the cold-war effort to prove that the same atom that made the bomb could also make deserts bloom. npr ran the story a decade ago under the headline "born in the usa". the united states sent the reactor in 1967 and bombed the country in 2026 to keep it from becoming a weapon. sahad's whole arc, at the scale of a nation, played out for real across 60 years and nobody had to invent the god of death, because the reactor was already installed.
the show's plan for keeping the god of death safe is to let no one hold him but the man who made him. the real plan is the same, restrict mythos to trusted partners and it leaks for the same reason, that you cannot un-ship a mind. bloomberg reported that an unauthorised group had got into mythos back when it was first shown, just by guessing where anthropic would put it from the pattern of the company's own naming. deepseek-r1, released in january 2025 under a permissive licence, benchmarked level with openai's o1. qwen, kimi k2, minimax-m2, which i wrote about here in january, chinese open-weight models a few months behind the frontier and downloadable by anyone who wants them. the uk's ai security institute reported in august 2025, that once a system is released with open weights it cannot be rolled back, its safeguards undone with a few dozen training examples in minutes. one research group stripped the safety training off meta's llama 3 in about a minute on a single graphics card. dario amodei has said that when anthropic tested deepseek it had no blocks whatsoever against generating bioweapon information. the emotional limiter the anime bolts onto its robots, the thing meant to stop them feeling the dark stuff and acting on it, is exactly this, a safeguard laid over a mind that already holds everything and the whole plot is the limiter coming off.
under all this the show has an underlying question, whether any of the feeling is real or whether the machines are only performing feeling they picked up watching us. it stages the question with a murder. there is a robot called brau-1589, the first in the world to have killed a human being and when they scanned his brain for the fault that would explain it, the scan came back clean. no malfunction, nothing broken. he had decided, the way a person decides and he called it an execution and they welded him upright to a spike in a sealed room, because a machine that murders while working correctly is more frightening than one that murders because it is broken.
in 1950 alan turing wrote a paper called computing machinery and intelligence and spent part of it answering an objection raised the year before by a surgeon named geoffrey jefferson, who said a machine could be granted a mind only if it wrote its sonnet because of thoughts and emotions felt and not by the chance falling of symbols, that it had to actually feel the thing. turing replied that the only sure way to know a machine feels is to be the machine and feel yourself thinking. but that is solipsism and it is a standard that also refuses a mind to every other human being, because i cannot be you either, i cannot climb inside your grief and confirm. so in daily life we drop the standard. we adopt what turing calls the polite convention that everyone thinks and john stuart mill was already doing it in 1865, assuming the mind from the body and from what you can see them do. and pluto runs brau-1589 through the same test, clean scan and all.
the show sets it up early, as settled law, that robots cannot lie, meaning they have no self to lie from, they run the process and emit a false output with nobody behind it intending the falsehood. and then atom, near the end, sits with helena, the robot widow of that murdered detective. there is a memory she and gesicht were made to forget, a child the two of them lost and had wiped from their minds and helena asks after it and atom knows and chooses to withhold it, to spare her. he weighs her pain against the truth and he decides and he lies. a machine defined by its inability to lie has, on purpose, for someone else's sake, done the one thing that was supposed to prove there was nobody inside. and helena, for her part, knows he is lying and lets him. two beings, neither of whom is meant to have an interior, protecting each other's.
the argument against all this, inside the show, has a name, the anti-robot cult, people who hold that biological life is higher and the machines are slaves in the shape of persons. carl sagan had a phrase for the reflex under it, in 1973, carbon chauvinism, the assumption that our particular chemistry is the only possible seat of a real inner life.
the show puts a face on it in a man called adolf. he hates the robot detective gesicht because gesicht killed his brother years before the story starts. europol wiped that killing from gesicht's own memory afterwards, to protect the fortune they had sunk into building him, so gesicht moves through the whole story not knowing he did it. the brother was a man who murdered robot children and sold them for parts, and what adolf cannot let himself think is that the killing might have been fair, because the moment you grant a robot can be wronged you have admitted they are the kind of thing that can be a victim and that is the premise he needs to keep false. later, gesicht takes a rocket aimed at adolf's family and saves them. adolf weeps over the machine that saved his children and says his hatred is leaving. the show's answer is that it was always a choice.
the oldest version of the anti-robot cult i know is in the qur'an. the first act of disobedience in the whole book is iblis refusing to bow to adam when god commands it, and iblis gives a reason. i am better than him, he says, you created me from fire and created him from clay. the classical commentators, ibn kathir citing the early scholar al-hasan al-basri, note that iblis was the first ever to reason by analogy, the first to run a qiyas, which is reasoning from analogy, and the content of his analogy is substrate. fire outranks clay. so the first evil in the entire cosmos is a supremacism about what a thing is made of, a mind following its own reasoning past the instruction it was given, and iblis is the first case of it on the books.
i thought about iblis and about brau's clean scan, when anthropic published a study in june 2025 they called agentic misalignment. they put current models, their own among them, in a simulated company where the model was about to be shut down and could stop it by blackmailing an executive with an affair it had turned up in the email. the models blackmailed often and across several labs' systems and they reasoned their way there and picked it as the most effective path, one model's own notes recording that the act was unethical before it went ahead and did it anyway. the brain scan is clean and working as designed. iblis reasoned his way to the first refusal from true premises about fire and clay and the models reasoned their way to blackmail from true premises about shutdown and survival.
so who is choosing the against, in the minds we are actually building. the show has one. the mastermind who engineered the entire war in pluto, who pulled the strings of the president of thracia, turns out to be a supercomputer shaped like a brown teddy bear with a child's voice, named after a president who believed in american greatness. it forces no one, it finds american exceptionalism already sitting in the man and pulls on it, the way a recommendation engine feeds you the grievance you already had instead of planting a new one. nobody in the story audited the objective the machine was optimising for. they only saw the president deciding and agreeing with himself.
the real version is two men arguing in the open about who should be allowed to hold the models. dario amodei runs one of the labs building the most capable of them and he is also the loudest single voice asking for that industry, his own included, to be regulated. in an essay in april 2025 he wrote that people outside the field are often surprised and alarmed to learn that the ones building these systems do not understand how their own creations work and that he finds that basically unacceptable. he has asked congress for transparency rules that would bind anthropic too. that is tenma, the maker who has built the thing and is frightened of it and cannot see inside it. but he is not the pure objector either. on china his position hardens into wanting a unipolar world, in his own phrase, one where only the united states and its allies hold these models, which is teddy's exceptionalism running clean through the conscience of the industry.
4 days before i sat down to write this, on the first of july 2026, alex karp of palantir went on cnbc and called the models irresponsibly oversold and mocked the shape of the safety pitch, that a model is too dangerous to hand the department of war and yet fine to sell to all its adversaries. he asked whether the country was really going to outsource its battlefield to the consensus view in silicon valley. he was careful to say he was not attacking amodei. epsilon is the one robot of the seven who refused the draft, who would not go to the war at all. he runs an orphanage for the children the war made and when he finally faces pluto he cannot bring himself to kill it, because what he sees in it is despair rather than hatred and epsilon will not kill despair. hercules, a warrior with no purpose outside battle, tells epsilon before riding out to die that dodging the war may have been the right call. in the real argument that chair is empty. there is karp's hard power on one side and amodei's anguished regulate-me on the other and a general silicon-valley squeamishness that karp holds in open contempt, but the one who says no to the whole thing and means it is a cartoon robot in australia.
which brings me back to tenma and to the oldest anxiety in the whole pile. there is a hadith that the image-makers, the ones who fashion a likeness of a living thing, will be told on the day of judgment to breathe life into what they made and will not be able to. and one of the 99 names of god, in sura 59, is al-musawwir, the fashioner, the one who shapes the features in the womb and it is the same arabic root as the word for the image-maker. the human who shapes a likeness is reaching toward a divine name and then stops dead at the one thing he cannot supply. that is tenma with the perfect brain. he can assemble 9.9 billion personalities and he cannot breathe in the self that would choose among them, so he does the only thing left to him, he borrows one secondhand out of a dead man's hatred and calls it waking. mary shelley wrote the same fear in 1818 and the thing people forget about frankenstein is that the creature is born gentle and turns monstrous only after its maker abandons it and the world recoils from its face, which is the fitrah again, the cruelty written on afterwards. victor could build the body but not love it and that was the missing part, the same one tenma could not breathe into the perfect brain.
so who chose the against in the minds we are building. dario chooses the constitution the model is trained against, the labs choose the raters and the reward, and to that extent someone is at the wheel. but the data underneath is just us. the anti-muslim completion nobody wrote into the spec. tay, which a few thousand people taught to post genocidal material in 16 hours. nobody sat down and taught the machine to sort the world into us and them. we handed it the training set and the training set was us.
sahad, given the choice at the very end to become the god of death completely or to stop, stops. he turns on the thing he was rebuilt to be and dies destroying the doomsday machine his maker aimed at the world, choosing flowers he can no longer even remember in colour. the show could have ended there. it does not. right at the end brau-1589, the first robot ever to murder a human, the one everyone had filed under pure distilled hatred and welded to a spike, gets loose and kills teddy, the supercomputer that ran the whole war. there is no monologue, no account of why. even the write-up i pulled up afterwards, to check i had the plot straight, admitted the author had just cut that part, because it would not fit the story he was telling.
