The boring systems did a lot of good work
Before LLMs turned up, engineers were already automating serious work: routing alerts, scheduling jobs, validating data, provisioning access, reconciling payments, and escalating a person when the rules ran out.
Those systems were not lesser because they were boring. They were useful because they were explicit. Given the same input and state, a deterministic program follows the same path. You can test it, replay it, and explain why it made a decision.
That is our starting point.
The Workbench situation
Our fictional online shop has an incident: checkout failures rose after a deployment. The first version of the Engineering Troubleshooting Workbench does not guess. It takes known signals and applies known rules.
if checkout errors are high and a deploy happened recently:
label the incident “possible deployment regression”
ask an engineer to inspect the deployment
else:
follow the normal checkout runbook
That is automation. It is also a perfectly respectable design.
What automation needs
Reliable automation is usually made of five things:
- Inputs — alerts, forms, events, API responses, schedules.
- Rules — conditions, validation and policy.
- State — what has already happened and what is still in progress.
- Effects — a ticket created, a message sent, a task queued, or an approval requested.
- Evidence — enough history for a human to reconstruct the decision.
When somebody tells you their system is “agentic”, ask where these five things live. A model can help with one part of the work. It does not make the other parts disappear.
Set up the Workbench
We will use the Workbench throughout this journey. It is a small local Python application with synthetic incidents only—no cloud account, API key, Docker container or production access is needed.
You need Python 3.10 or newer and Git. Open a terminal and check what is already on your machine:
python3 --version
git --version
On Windows PowerShell, use this Python command instead:
py --version
git --version
If Python reports a version below 3.10 or is not found, install a current version from python.org/downloads. If Git is not found, install it from git-scm.com/downloads. Once both commands work, clone the Workbench:
git clone https://github.com/hmohamedansari/engineering-troubleshooting-workbench.git
cd engineering-troubleshooting-workbench
macOS and Linux
Create an isolated Python environment in this project, activate it, then install the Workbench and its tests:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
Windows PowerShell
Create and activate the same isolated environment, then install the project:
py -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
If PowerShell says script execution is disabled, run the following command once in that same PowerShell window, then activate the environment again:
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
Run and inspect it
Start with the terminal view:
workbench
It prints the incident state, the evidence it collected, the selected route, why that route was selected, and the next safe action. Nothing is hidden behind a model call.
Then open the visual view:
python -m streamlit run app.py
Streamlit prints a local address—normally http://localhost:8501. Open it in your browser. Choose Checkout regression and inspect the evidence, the two rule inputs, the route, the incident state and the event timeline. Keep this terminal open while you use the browser.
Try it
The Checkout regression scenario observes an 8.2% checkout error rate. In the browser sidebar, raise High checkout error rate from 5% to 9%. The route moves from possible-deployment-regression to normal-observation. Set it back to 5% and the original route returns.
Nothing mysterious happened: you changed one visible rule, so the visible decision changed. Run pytest afterwards to see the rules this checkpoint protects.
Keep this with you
The question is not “Can an LLM do this?” Start with: “Can a clear rule do this reliably?” If yes, use the rule. Save uncertainty for work that actually needs it.