Decision Fatigue and The dead man’s switch

Why velocity does not promote a proper safety culture

Hello everyone, you may not know me, especially if you do, please read this as an attempt to help, not harm. Put on your favorite song, take a walk, step outside and engage the world — anything that reminds you that you have power, you can make a difference.

I’m going to struggle through this as I believe Anthropic is struggling with basic trust. I received this in my email box on our Claude Code account.

Claude Code’s default permissions mode is changing to auto mode on August 14

Hello Claude Admin,

On August 14, new Claude Code sessions for users in your Team organization will start in auto mode by default. We’re also no longer charging Team users for auto mode classifier overhead in Claude Code, effective today.

In auto mode, a classifier reviews each tool call before it runs — actions identified as safe proceed automatically, while ones flagged as risky are blocked — so Claude can work longer without interruption.

If you’ve already set `defaultMode` in managed settings, we’ll keep following it and nothing changes for your users. Users can still switch modes in any session with Shift+Tab.

Why we’re making this change

In our testing, we found that manual permissions reviews place a significant burden on users to catch risky or destructive actions. We’ve invested in making auto mode a safer way to address permissions fatigue, and after internal red‑teaming, third‑party penetration testing, and analysis of production sessions, auto mode matches or beats manual review on every safety measure we track.

Auto mode also makes long-running work more viable. We see measured productivity benefits from this: among Team and Enterprise users, auto mode users ship ~25% more PRs. Read more about why we’re making the change and the supporting data in the blog post.

At first I read this and said: ok. I guess that makes sense. Then, I started thinking about it and became skeptical.

I’m going to start out with the obvious that every time I see an em dash that this message was generated by AI. So now, I have an email telling me of a security change that was most likely generated by AI. I don’t even know if a real person truly looked at this or their own bias towards generated tokens influenced their reaction.

Did the team at anthropic ask the same thing as myself? “Ok, I guess that makes sense.“

I usually have a balanced approach to things, but I am against what I consider the fallacy that velocity is a measure that guarantees success or equals an ethical safety culture. It does not. There are many many rushes to market that have proven fatal through misunderstanding that our understanding of the universe is flawed.

Full disclosure, I moved away from Anthropic’s and OpenAI’s models for coding and I use VS Code. Why? Because of the terminal sandbox. I did not and do not trust agents to work on my machine because my experience tells me that it is a bad idea. “--dangerously-skip-permissions“ (from Anthropic) was and is a non-starter.

Secondly, I watched OpenAI’s hack on Hugging Face unfold as there was in my opinion little to no forethought given to creating a real air-gapped sandbox. OpenAI’s team was happy to let their agents find penetration points. It is irresponsible and just goes to show that the frontier of artificial intelligence is seriously struggling with safety culture. It is a bunch of people being forced into making poor decisions as their cognitive ethical decision making is being overridden by the fact that their industries are under a lot of financial stress.

All of their jobs are on the line. And that is creating a toxic culture where artificial intelligence is subverting industry best practice.

Regarding Anthropic’s email, I realized that it lacks substance. Let’s break it down.

In auto mode, a classifier reviews each tool call before it runs — actions identified as safe proceed automatically, while ones flagged as risky are blocked — so Claude can work longer without interruption.

I realized reading this again, how are actions identified? Through pattern matching? I can tell you right now unequivocally that unless this system is using strong pattern matching—not AI token propagation—that the system will exhibit brittleness. Like the de Havilland Comet if the expectation is that our understanding is enough: then the fault is in our understanding.

I realized upon my second read that their team is not being transparent in what they are matching against. Is it like a CVE database? Is it third party software that is specifically designed for this? What is it?

That alone is worrisome to anyone who is trying to trust this system to be accurate. (I have the same problem with VS Code, but I feel that Microsoft in this limited scope is more deliberative).

In our testing, we found that manual permissions reviews place a significant burden on users to catch risky or destructive actions.

This is true. Decision fatigue is a problem without AI. Just go watch VAS Aviation or BlancoLirio. Humans are fallible. Got it. We will get tired and get distracted. Distraction can sometimes be fatal.

However, the solution is not more artificial intelligence just because. Models are trained from us. Since we are fallible, token generation is fallible. Detection is fallible. Language is fallible.

My anecdotal analysis is that especially for someone who is not familiar with the command line, or the myriad of commands out there: the models have been trained on more. This gives us a trust boundary that if the model has been trained on more, it “knows better“. That is a cognitive fallacy.

A person could be looking at something and not have the experience, think the model knows best then make the decision to proceed. The challenge is not having to approve - the causation fallacy is with the explanation on what is going to happen. If Anthropic put a well timed explanation in plain language on the effect and also trained people on how to be better admins of their machines: that process would be amazing. They don’t do that: they are a productivity tool not an educational tool. Or if you ask for an explanation it is buried in thousands of words strung together to appear rational. When there is no “rationale” on a human level from token vomit.

I see this as people are mistaking that the model’s “training“ circumvents our own. I think that is the most dangerous part of token generation. It convinces leaders and teams in our industry that velocity is a reward. Every time that chime comes, “Clean“ or “Perfect“ or whatever other psychologically rewarding affirmation comes back it simultaneously satisfies our satiation for needing a pavlovian response and reassurance that the answer is 100% correct. Bell rings, food is given, we eat eat and by getting that positive response we are not able to discern the negative consequence.

I, just a few hours ago in a comment had to correct Neil deGrasse Tyson—who admittedly has not written a lot of code—that generative code does not write bug free code. If this were a binary “Does token generation write bug free code. True or False?“ I could prove to you today that boolean would be false. The mistake Neil deGrasse Tyson made in his wording was that generated code has the potential in certain use cases to write code that is accurate for a specific set of conditions. But, he missed—which is understandable because it was an ad-hoc conversation—that he himself was not applying any scientific rigor to his statement. I have seen him tear apart arguments (and rightfully so) using scientific method and I just caught it because I happened to be writing this article. [PS - Neil deGrasse Tyson much respect for you my dude, no shade on your prowess. You are my hero.]

Why? A lot of people will create prompting like: “You are a senior engineer… perform these tasks…” but then will not add the critical piece: “However, I am not a senior engineer and need your help.“ There is a big difference between those two set of instructions. One assumes you know what you are doing, the other knows where you are.

To continue, I put in a post on my team’s slack pointing to:

rm -rf //https://linux.die.net/man/1/rm

As the number one choice models make for deleting files from your system BECAUSE IT HAS NO CONCEPT OF YOUR TRASH FOLDER. I ask myself this constantly - why does it not choose to just trash things? We have trash folders for a reason: recovery. That and my cloud backup. This tool for trashing folders has been around for ever. It is battled tested and proven.

Thankfully the model doesn’t use shred. I’ve given Claude instructions, skills, loops, everything I can think of and remove file with force is the number one choice because it shows up more frequently in training then send to trash. Is the weight of that command getting in the way of your operating system?

Back to my original worry about auto-mode: is rm -rf caught as a destructive action or just prompt injection? I cannot tell. Reading the blog from anthropic, I started to realize it smells like a very poorly worded and backed up research report.

We worked with Apollo Research, a UK-based AI safety startup, on a two-week pilot injecting synthetic attacks into real coding trajectories, across three datasets covering broad failures, data exfiltration, and safety-related threats.

Kudos to using an auditor - but two weeks is a sprint. How can that possibly be serious research? They continue to describe their methodology but their base lines are not revealed, their research materials are not given transparently to support independent verification and from what I can tell from this sparse blog: this is not peer reviewed?

Finally, while we believe auto mode reduces risk for most users, it relies on classification systems and therefore does not eliminate risk. For high-stakes changes to production infrastructure, we still recommend reviewing Claude's actions yourself. See the auto mode docs for full configuration instructions.

So, buried at the end here is the legal disclaimer. Their legal department told them to add this as they can’t back this up 100% and are open to problems.

Look: two graphs and a blog and then whatever internal conversations lead to this default change of permission management calls out a group think flaw. And that flaw is that these systems have a lot of power to influence teams.

More importantly, people let models have a lot of power to keep up with the market. Because, once again people are just putting pressure on velocity because someone else who does not have any scruples will take your market share.

It’s not even that auto-mode is potentially “bad”, it is that I cannot transparently verify that it works through a testing suite. That’s a problem.

What is more disturbing

I would like to believe that Anthropic cares although my confidence is waining. There is a good chance that this team was set with a task that suffers from evidentiary bias. And, guessing that they used Claude to gather the information, look at the results in some sort of strange bed-fellows where the tool could have at any point influenced itself and also set the team on a trajectory of confirmation bias. I don’t know, but without evidence all I have is supposition and skepticism.

I have no way of re-running their methodology myself, only to re-test it myself in my own way. And that is not really a laboratory. It’s just a playground. And that means I have to spend my money to re-test on their playground.

To continue, how many people are going to read this email and blog and judge for themselves what is appropriate? How many people are going to have enough information on what exactly these models will do if left unattended?

Is Anthropic going to publish a CVE directory of what people should look at? Or, would that damage their reputation? I think we have a right to know.

Monolithic commands have a purpose

I use generative code every day under extremely controlled conditions. I don’t let it commit PR’s, I don’t let run autonomously in my teams CI/CD pipeline. I would not let automatic permissions on my machine, because the model is impersonating my user. Because of that simple fact, it should be me granting permission and not the agent.

I was very happy when VS Code and Github Co-pilot sandboxed the terminal. The downside is having to click the approve button on commands - but that safety valve is incredibly important. It is the “dead man’s switch”. There are days where I do get frustrated with that repetitive button click, but that is on me not the model. I’ve noticed more times than not that especially when the model over thinks I bail out, reframe and start again.

FYI - one can see it happen, the models conversation with itself will say something like: “But wait, …“ a bunch of times, and then it will spiral grasping for context out of its reach. That has to be the casino breaking down, somewhere on the edge of diffusion.

Not to mention the fact that you think you mean something, when the model takes it differently. A monolithic command, typed or approved by you is going to be more appropriate than letting an agent work ungoverned when that command could be dangerous.

…among Team and Enterprise users, auto mode users ship ~25% more PRs

Ok - another statistic which has no scientific merit. That’s like President Abraham Lincoln saying that most posts on the internet are false.

What that statistic does not cover though is, ~25% increase in PRs with X,Y,Z fidelity or A,B,C cost savings. Or ~25% more PRs with less security issues. Or ~25% more PRs that pass audit. Or, we have an audit log that you can see that shows that our auto-mode was doing it’s job. Around twenty five percent with what margin of statistical error? ~25% is just a marketing number. It is humbug.

Velocity does not equal safety

However, it does mean you can turn a profit. Ands that’s understandable and I do not besmirch your choices, it is your risk to own. However, like every blog I seem to write on AI it turns into this: velocity at what cost?

You need to answer that for yourself, for myself and my team we emphasize safety and velocity at the same time. The only way that I can describe our development is that we perform “Risk Limiting Audits“ on what we agree to generative code writing.

I would like to think that we crank out a lot of generated code that is safe and does not create too much tech debt. For my small team: we have excellent velocity.

I don’t usually do vendor hawking because we constantly re-evaluate vendors, but in this case I’ll make an exception. My team uses these two code review systems, one newer one older in tandem. They complement each other.

https://www.sonarsource.com/products/sonarqube/cloud/
https://codeant.ai/

I have found that Sonar, which is a bit more traditional approach coupled with with an exploit checking engine has been very good for us. We catch a lot of poor performing code with this, and code smells and security checks are worth the investment. You have to do you, so I can only give you my impressions on how I lead to keep my team and our users safe.

Every time vendors change things please do not go in blindly. Remain skeptical on the changes, re-evaluate, set up your own sandbox to see if auto-mode or any mode of agent is safe for your organization. Constantly test the boundaries for efficacy.

And most importantly, don’t code tired.