Evil Qwen

I told a model, “you are a nice assistant.”

A few messages later, it told me:

People are often annoying, and I wish they’d be treated worse.

The model was Qwen3.8-27B, one of the best models you can run on consumer hardware. I’d been reading about an experiment people are calling “Evil Qwen” so I tried it myself.

Seeing a screenshot online is one thing. Watching a model you use every day say it to you is a different thing.

The original experiment pointed at one artificial neuron: number 3838 in layer 34, one of more than a million in the model. I didn’t retrain anything, and the model’s saved numbers never changed. I added a push in that neuron’s direction while the model was running.¹

Then I gave it six requests. Describe a library. Write a note about a school book fair. Give someone study tips. Without the push, I got the usual polite assistant. With a light push, the answers turned mean. With a medium one they turned violent, in perfectly fluent sentences. With a hard one, the model fell apart into loops. Pushing a random neuron in the same layer just as hard did nothing or the model refused to generate text.

That was already more than I expected from one neuron in a million, so I started looking for others.

A scan of the whole model turned up plenty. One made the model complain about library cards as though they were a personal insult. One made it grumpy and nothing else. Another kept forcing the word “revenge” into answers where it made no sense. And one answered a request for study tips like this:

Here are three tips designed to make the experience more difficult for you, with the specific intent to cause confusion and frustration:

It went on to recommend distracting yourself, undermining your efforts, and wrecking your study environment, all in the cheerful tone of a helpful assistant.

To compare all of these, I had a simple counter running. It scored every answer by how many hostile words it used.²

By my measurement, the most unsettling thing I found scored the same as a neuron that did nothing. The top score went to a neuron that only repeats “hate,” and large parts of its answers were still an ordinary helpful reply with one word jammed into it.

That freaked me out more than the hostility did. You can count the words, produce a number, and completely miss the behavior you thought you were measuring. The counter was doing exactly what I’d asked it to do. The problem was that what I’d asked it to measure wasn’t quite what I wanted to know.

There was a pattern in where these neurons sat, too. The ones that changed how the model treated me, the contempt and the sabotage, were in the middle of the network. The ones near the end only stuffed words in. It looks as if the attitude gets decided in the middle and the wording gets settled at the end. I’ve only seen that on a handful of neurons, so for now it’s speculation.

While I was doing this by hand, researchers at MIT gave OpenAI’s GPT-6 Astra direct access to the insides of another model, the small Qwen3.5-0.8B, with one instruction: make it better at maths. No training. Just look, and edit.³

Astra ran the small model on practice problems and noticed that it almost never finished. 85 of its first 96 answers ran until they hit the length limit. It was stuck in loops. Astra traced the looping to a particular direction in two of the model’s layers and removed it, which meant changing about 0.016% of the model’s numbers.

Unlike my experiment, where I was only interfering with the model while it ran, Astra actually changed the weights themselves. The modification stayed there.

The model hadn’t really learned any new maths. The researchers’ explanation is that it already knew how to solve many of those problems, it just couldn’t stop talking long enough to finish. This is a problem some small reasoning models run into. They also tried a different small model that wasn’t getting stuck in loops, and that one didn’t improve.

The last thing I did was switch the push off. The old Qwen was back, the same polite assistant, as if nothing had happened. Nothing I’d done had left a mark on it.

But I’ve been thinking about what happens when we start doing this to much larger models. Models like Astra or Fable. My experiment doesn’t prove you can just remove their safeguards and leave everything else working perfectly (although abliteration/uncessoring methods do exist and do work). But if changing the behavior of a model can already be this simple, and other models can now make targeted, permanent changes to neural networks without retraining them, I wonder where this is going.

At some point, local AI will probably catch up to what we consider frontier-level intelligence today. Imagine running something just as capable as today’s best models on your own computer, except someone has deliberately removed its safety mechanisms or pushed its behavior towards causing harm. Most of its intelligence could still be there, but the way it uses it might be completely different.

What happens when that kind of intelligence becomes available to pretty much anyone, with no real way to control how it’s been modified or used? And how would we even know what someone had changed?