AI handles incidents, engineers lose touch with their systems
344 by sylvainkalache · · sylvainkalache.com · 302 comments
Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs.
Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc.
Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it.
Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.
I find the use of AI like quicksand.
The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.
The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started).
Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.
I see aviation sentiment raised from time to time. In aviation failure is catastrophic, and systems you operate do not change on the fly (pun intended).
You can probably drill SREs that way but you are only teaching them how to _react_ efficiently, not how to _fix_ unique unicorn root causes of these failures. Aviation analogy would be an airplane engineer that tries to drill himself for in-flight engine failures AND engine debugging / engine redesign both at the same time.
It never made sense in software engineering, and never will. Minute spent on drills is a minute better spent on reworking codebase to reduce changes of future incidents. This post is very SRE centric.
Code too.
I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past.
Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago.
This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.
I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system.
They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI.
You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.
So looks like you pay AI to resolve incidents and then pay money and time for engineers to get training on synthetic incident resolutions.
The comparison with Pilots is bit disjoint one cause the domain varies a lot for every company and product. Coming up with synthetic simulations within every domain is like paying money twice for the same thing, why not let the engineers to handle the real incidents in the first place itself. In fact why not spend some more thought into building better systems.
If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.
If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.
This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.
There is only so much need for taste and judgement, before even that has been incorporated into the models.
There's an interesting dynamic, too. Even if an engineer reads the output of the AI and understands the root cause of the problems and how to diagnose the incident, somehow it's hard for them to internalize the learning and apply it next time to a new incident. As a result, the engineer loses touch with the system anyway.
It looks like our brains somehow have to experience the failures during a diagnosis and in gemerak perform this kind of pathfinding by themselves to truly understand the system. I don't know if this has to do with how our brains actually learn.
This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.
Its true we will lose the skills, but so far the LLMs are more than picking up the slack when wielded competently.
They _regularly_ go above and beyond when troubleshooting and frequently in 1/2 - 1/1000th of the time.
I remember spending entire days troubleshooting in the before times. Now it's like 30 minutes, tops, on literally any issue.
This is what progress looks like. We used to do a thing and now we don't.
This is the same problem as the "AI drives the car until it can't" situation. Feel free to doze off so long as you can wake up and instantly have world class racecar "save the situation" reflexes. Hope isn't a strategy but that's what all of this feels like.
This is the same challenge aviation had with an over-reliance on automation. You end up with pilots that can’t fly the plane when it really matters. Look up the Asiana crash at SFO which is a lesson in what will happen to engineering orgs that over-rely on AI.
Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong.
Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.
If an RPA team ever gets direct access to a production database in your company, look for a new job.
We have been running Agents on infrastructure and letting to create resources, scale up and down, security scans etc.
I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes.
We recently had a situation, where in with our internal platform and claude we recreated everything in minutes.
Management in the end cares about the outcome and not how the meat is made.
I like the idea of simulations - maybe not in the flight simulator sense of a fixed rhythm, but more in the training in using the tools quickly sense, like a chaos monkey in your log search. My most memorable exam was Certified Kubernetes Administrator (CKA), the hands-on simulation part was exhilarating.
Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?
[comment deleted]
I agree with the sentiment but unless it leads to better financial outcomes for them to A) have engineers on staff and B) have them know their systems then enterprise will continue looking for reasons to shed all the engineers and just leave everything to AI. Anecdotally I think most companies are still looking for ways for AI to help them shed headcount so raising this alarm doesn't mean anything to the decision makers at the top.
Wasn't that the idea? Withe cars on roads, didn't we literally lose the touch with the ground? I guess the desire was to "lose" touch with all dirty and hard work areas.
One time I had to do a system optimisation to increase the throughput of messages and was using Claude with Datadog and Couldwatch MCPs to figure out the bottlenecks by running load tests. it was Opus 4.8 and one of the most frustrating interactions with Claude I ever had.
It was just making up random stuff about AWS and system resource limitations and when I was asking for the source like from where it got that info, it was like, “I’m sorry I prematurely concluded that without checking sources”.
I would never trust an Agent to resolve incidents ever
Suppose "Reflections on Trusting Trust" is not the naive rantings of some obsolete dinosaur.
Why does it not apply?
This has already happened in other areas. I'm old enough to remember when customer service people spent all day handling customer problems themselves, which gave them the understanding needed to solve more unusual problems.
These days they do what the computer tells them. Even if they could solve the problem they don't have the agency. They can only select from the options they are given.
In software we still have the agency but we are giving away the understanding. The agency will follow.
Reminds me of a story about Ericsson engineers missing vital troubleshooting practice when Erlang turned out to be too reliable.
When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust.
One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time.
It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...
I’m not sure who will be solving these incidents if 99% of incidents will be “solved” by LLMs. If I’m called once a year my daily rate will be my yearly rate?
I still use LLMs, but strictly for sanity checks or quick summaries. Trusting them blindly to debug complex systems often leads to a rabbit hole.
"On a silver platter" lol The old phenomenon: give the cashier at the convenience store exact change and they will be like, d-uh. Because automated cashiering.
I detest these doom and gloom articles.
We’re entering a new phase of software development, and with every phase there are new challenges.
Some questions the author might want to first answer: 1. If AI is constantly reacting to smaller incidents, is it surfacing larger issues in your codebase and architecture? 2. What kind of new telemetry do we have to build? 3. How do we build new deployment systems that help us validate fixes without requiring hours to go through traditional CI/CD systems
I’m old enough to remember the days when engineers would monkey patch their code on live servers with additional logging and metrics to learn about failure modes during incidents.
Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me.
Depending on what your goals as the author are, you may or may not want that.
Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.
[comment deleted]
This article leads up to a hypothesis presented without evidence.
Yes, it's clear that if AI agents handle routine issues then only the most complex issues will be handled by humans. In no way does it follow that humans will be less prepared to handle those incidents than they are now; by definition, this class of problems have no rote solutions. Each issue of this type already requires deep system knowledge to remediate, and still will. Software is not aviation.
What we are seeing here is an instance of the more general trend where experts are still required to operate complex systems, but AI is destroying the career path that creates those experts. That's what we should be worrying about, not that people are going to be spending less time rebooting boxes. This isn't something you can fix with a "simulator."
Detection rots first when an agent owns the runbook...you lose the skepticism, not the fix-velocity.
This sounds like the next iteration of "nobody understands their runtime environment these days", which I've been saying for many years.
You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.
The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not.
An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.
I feel a lot of comments here are missing the forest for the trees.
We do not yet have the next generation systems that will manage AI creation and maintenance of systems.
Humans have been making spaghetti code systems and maintaining them poorly for years. Best practices developed… eventually. But certainly not in the 70s and 80s. Spaghetti was the norm for quite some time.
The development paradigm has changed. Forever. You can’t expect yesterday’s tools for managing software development to succeed at this point. We’re still roughly on year one of this transformation.
The new bottlenecks are creating and enforcing boundaries in the code, identifying level of risk within a boundary, subjecting high risk areas to more intense human review and architecture reworks, and so so much more manual testing.
We need a new language for high level development that focuses on architectural constraints. We need analysis software that helps draw boundaries, identify what needs extended human attention, and helps us map and understand a rapidly developing code base. We need to standardize on the use of frameworks and languages like we have on assembly variants.
I’ll bring up a hacker news trope here. NaughtyDog’s GOAL was an amazing system that was a product of an underdeveloped ecosystem for game development at the time. They used lisp and assembly because you still had to write assembly for performance reasons then.
We are in a very similar period. The ecosystem is underdeveloped. We should be looking for new languages and tools to manage this.
CNC machines used to run from punch cards and then hand written NC code. Now we have advanced CAM software. (Which innovative people are actually running with LLMs!)
The way forward is clear. Improve the AI such that the hardest incidents can be handled by AI alone. Whether we want this or not is independent of the trendline which points to this future regardless.
Imagine a future where humanity understands none of the underlying technology and AI just babysits us on a daily basis. Wall-e.
Programming will devolve into unintelligible anti patterns and will be written in convoluted languages humanity doesn’t understand. But none of this matters because AI will be handling all of it.
Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.
same irony hits agentic remediation. ai clears routine incidents, humans inherit the sev0 with zero reps. shadow mode is the simulator.
I run an IR company focused on specific security response emergencies.
AI use across my company is variable and I don’t allow any enforcement of mandatory using or not using of it.
I view and tell my employees to view their skills as that of a high-performing athlete. Consistent training of the basics, rapid fire what-if testing, weekly scenarios are all part of the upkeep.
In short, this is what it takes to keep up the skills no matter if you do or don’t use AI and I can imagine the degradation is much faster if you don’t train AND delegate all of your activity to AI.
I recently had some issues with a game not working well on my Intel Arc B580. OpenAI (Anthropic's models definitely could too) ran some profiling inside of Windows, captured a few GB of data, sifted through it, identified the guilty .dll's and approximately where the issue was and traced it down to a single user-toggelable Unreal Engine 5 launch flag, which resolved it.
I feel like AI could easily do the same for servers and various software, BUT you gotta ground it in actual data so it can't make the shit up and just spew out garbage (if you tell it to come up with a potential answer it will even if it doesn't have the proper data to be "sure" of it).
Prediction: this won’t happen. The abstraction will be good enough and people will need to know only as much as they need to know- things will stabilise at the equilibrium.
Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM.
Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short.
There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.
[comment deleted]
I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.
I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.