Ajay Walia

The Minion Who Wanted a Touchscreen

Ajay Walia — Sat, 06 Jun 2026 00:00:00 +0000

Chapter 1 — The Problem

We live in a world of glass that responds to us. Tap an iPhone, it opens. Swipe an iPad, it scrolls.

But when I reach out to my MacBook display? Nothing. Just fingerprints.

The interaction model is broken. We expect a simple flow:Finger ↓Screen ↓Content Moves

But reality on a Mac is:Finger ↓Screen ↓Nothing Happens

Chapter 2 — The Crazy Idea

I didn’t want to buy an external touchscreen, and I certainly didn’t want a clunky piece of hardware taped to my bezel. I just wanted my Mac to understand when my finger was moving across it.

What if we could create an invisible sheet of glass in front of the display?

Chapter 3 — Building an Invisible Touchscreen

To make this work without lag, it couldn’t be a Python script running in a terminal. It had to be a native macOS menu bar app, tapping directly into the Apple Neural Engine. Here is how the v2 architecture flows:

To explain this, let’s meet the engineering team:

Camera Manager Captures anAVCaptureSession at a strict 30/60 fps with actor isolation.

Vision Pipeline UsesVNDetectHumanHandPoseRequest to identify the index finger in the frame.

Hand Tracker Applies a centroid-based lock (0.15 threshold) so the system doesn’t jump between hands.

Scroll Engine A GestureStateMachine (Idle → Dwelling → TouchActive → Releasing) that posts FPS-normalised scroll events directly into the macOS HID system.

Chapter 4 — Why Gesture Control Is Wrong

The first instinct when building camera control is to use gestures.

If I have to learn sign language to read an article, the tool failed. I didn’t want gesture recognition. I wanteddirect manipulation.

With Glide:

Point finger.
Touch page.
Move page.

Chapter 5 — The UX Rabbit Hole

If you’ve ever used a bad gesture system, you know the feeling.

Imagine hovering your hand in mid-air. In a typical gesture system, you enter a “touch zone” and then are forced to wait. A mandatory 200-millisecond delay kicks in just to confirm your intent before the system finally registers the action. That tiny fraction of a second feels sluggish, unnatural, and deeply frustrating.

Now imagine a true physical touchscreen. The moment your finger touches the glass, the response is instant. No waiting. No unnatural pausing. The digital content tracks perfectly and immediately with your physical movement.

The difference between a gimmick and a tool is latency.

Chapter 6 — The Touch Plane

To make it feel like a touchscreen, we had to invent a virtual screen floating exactly 10 inches in front of the actual screen.

Imagine a strict, invisible boundary hovering parallel to your MacBook display. Until your finger crosses that exact depth threshold, the system entirely ignores your movements. But the moment your fingertip pierces that invisible layer, the interface wakes up—locking onto your finger’s precise coordinates and translating them into immediate, pixel-perfect scrolling.

Chapter 7 — The Tech Stack

To run a continuous computer vision pipeline without setting a MacBook on fire, the tech stack had to be heavily optimized.

Why Swift?

Using Swift gave us direct access to native macOS APIs and the Vision Framework. It meant better latency, memory safety, and lower power consumption since the Neural Engine does the heavy lifting.

Tech Stack Callout:

Language: Swift 5.9+ (Strict concurrency)
UI: SwiftUI
Computer Vision: Vision Framework
Event Injection: Quartz Event Services
Architecture: Menu Bar App
Platform: Apple Silicon (M-series)

Chapter 8 — The Real Challenge

The hardest problem wasn’t the computer vision or the math.

The hardest problem was comfort. Humans aren’t built to hold their arms out straight for eight hours a day.

Our Test Matrix (from the v1.0 Test Plan) reflected this. We had to pass:

Chapter 9 — What Success Looks Like

Final Section

The journey doesn’t stop at scrolling.

Today ↓ Scrolling
Tomorrow ↓ Zoom ↓ Click ↓ Window Management ↓ Presentations

At Aera, we are building a camera-native interaction layer for macOS. Glide is just the beginning.

An Honest Postscript — Into Cold Storage

I’ll be straight about where this actually ended: I left Glide midway. It’s in cold storage, not shipped.

A few things stacked up. Token limits and the sheer time Antigravity took to grind through each problem made every iteration expensive. And the LLM never managed the one breakthrough that mattered — the scrolling logic. We could detect the hand, lock onto the finger, calibrate the touch plane, and light up the debug HUD, but turning that into scrolling that felt genuinely like a touchscreen stayed just out of reach. At some point I chose to stop rather than keep forcing it.

But I don’t count it as wasted. I set out to treat this like a proper software-engineering project rather than a weekend hack — a real architecture, a menu-bar app instead of a terminal script, and detailed documentation that a serious project would keep: a maintained CHANGELOG, a test plan and test matrix, an architecture and a gesture spec. That discipline is the thing I’m taking with me. Even shelved, the project is fully legible — anyone (including future me) can pick it up and know exactly where it stands and why. That’s the real learning: good documentation is what lets a project survive being put down.

See it on the bench — the screenshots

Seven captioned frames from the build: the Xcode project and its CHANGELOG, the AI agent wrestling with the scroll logic, the full five-step onboarding flow, and the live debug HUD.

Open the screenshot gallery →

]]>

Cortex Swarm: Upgrading the Traditional IT Operations with Agentic AI

Ajay Walia — Sun, 24 May 2026 00:00:00 +0000

@import url('https://fonts.googleapis.com/css2?family=Noto+Serif:ital,wght@0,400;0,700;1,400&family=Noto+Sans:wght@300;400;600;700&display=swap'); .cs-article { --bg: #080810; --bg2: #0d0d1a; --bg3: #111122; --gold: #c9a227; --gold-light: #e8cc6a; --gold-dark: #7a5f0e; --blue: #1a2744; --blue2: #0f1d38; --cream: #e8dcc8; --muted: #9a8a6a; --blossom: #c4909a; --rule: rgba(201,162,39,0.22); font-family: 'Noto Sans', sans-serif; line-height: 1.75; color: var(--cream); background: var(--bg); padding: 2.5rem; border-radius: 16px; box-shadow: 0 4px 30px rgba(0,0,0,0.6); } .cs-article * { box-sizing: border-box; } .cs-section { padding: 56px 0; border-bottom: 1px solid var(--rule); } .cs-section:last-child { border-bottom: none; } .cs-label { font-size: .7rem; letter-spacing: .2em; text-transform: uppercase; color: var(--gold); font-weight: 600; margin-bottom: 10px; } .cs-article h2 { font-family: 'Noto Serif', serif; font-size: clamp(1.5rem, 3.5vw, 2.2rem); color: var(--gold-light); margin-bottom: 20px; margin-top: 0; } .cs-article h3 { font-family: 'Noto Serif', serif; font-size: 1.2rem; color: var(--gold-light); margin-bottom: 12px; margin-top: 0; } .cs-article p { margin-bottom: 18px; font-size: 1rem; color: var(--cream); opacity: 0.95; } .cs-article ul { list-style: none; padding: 0; margin-bottom: 18px; } .cs-article ul li { padding: 4px 0 4px 20px; position: relative; font-size: 1rem; } .cs-article ul li::before { content: "—"; position: absolute; left: 0; color: var(--gold); } .cs-statbar { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; background: var(--rule); border: 1px solid var(--rule); margin: 36px 0; border-radius: 4px; overflow: hidden; } .cs-stat { background: var(--bg2); padding: 20px 14px; text-align: center; } .cs-stat .num { font-family: 'Noto Serif', serif; font-size: 1.8rem; color: var(--gold); display: block; } .cs-stat .lbl { font-size: .68rem; letter-spacing: .08em; color: var(--muted); text-transform: uppercase; } .cs-visual { margin: 32px 0; border-radius: 6px; overflow-x: auto; overflow-y: hidden; border: 1px solid var(--rule); } .cs-visual svg { display: block; width: 100%; height: auto; } .cs-figure { margin: 32px 0; } .cs-figure img, .cs-figure video { display: block; width: 100%; height: auto; border-radius: 6px; border: 1px solid var(--rule); } .cs-figure figcaption { text-align: center; font-size: .78rem; color: var(--muted); font-style: italic; margin-top: 8px; } .cs-hero-video { position: relative; margin: 0 0 16px 0; border-radius: 8px; overflow: hidden; border: 1px solid var(--rule); box-shadow: 0 4px 20px rgba(0,0,0,.5); } .cs-hero-video video { display: block; width: 100%; height: auto; } .cs-hero-video .overlay { position: absolute; left: 0; right: 0; bottom: 0; padding: 40px 30px 48px; background: linear-gradient(transparent, rgba(8,8,16,.92) 30%); text-align: center; pointer-events: none; } .cs-hero-video .overlay .h-title { font-family: 'Noto Serif', serif; font-size: clamp(2.2rem, 5.5vw, 4.5rem); color: #fff; font-weight: 900; letter-spacing: 2px; } .cs-hero-video .overlay .h-sub { font-size: clamp(1rem, 2vw, 1.4rem); letter-spacing: .25em; color: var(--gold); font-weight: 700; margin-top: 14px; } .cs-hero-video .overlay .h-by { font-size: clamp(0.85rem, 1.5vw, 1.15rem); letter-spacing: .15em; color: var(--muted); margin-top: 18px; font-weight: 500; } .cs-cards { display: grid; grid-template-columns: repeat(auto-fit,minmax(230px,1fr)); gap: 14px; margin: 24px 0; } .cs-card { background: var(--bg2); border: 1px solid var(--rule); border-radius: 6px; padding: 20px 18px; } .cs-agent-row { display: grid; grid-template-columns: 150px 1fr; border: 1px solid var(--rule); border-radius: 6px; overflow: hidden; margin-bottom: 10px; background: var(--bg2); } .cs-agent-label { background: var(--blue2); padding: 16px 14px; border-right: 1px solid var(--rule); display: flex; flex-direction: column; justify-content: center; } .cs-agent-label .tier { font-size: .67rem; letter-spacing: .14em; color: var(--gold); text-transform: uppercase; margin-bottom: 3px; } .cs-agent-label .role { font-family: 'Noto Serif', serif; font-size: .95rem; color: #fff; } .cs-agent-label .replace { font-size: .67rem; color: var(--muted); margin-top: 5px; } .cs-agent-body { padding: 14px 18px; } .cs-agent-body .desc { font-size: .8rem; color: var(--muted); font-style: italic; margin-bottom: 8px; } .cs-tag-row { display: flex; flex-wrap: wrap; gap: 5px; margin-top: 6px; } .cs-tag { font-size: .67rem; padding: 2px 8px; border-radius: 20px; background: rgba(201,162,39,.1); border: 1px solid var(--gold-dark); color: var(--gold-light); } .cs-timeline { position: relative; margin: 24px 0; } .cs-timeline::before { content: ''; position: absolute; left: 19px; top: 0; bottom: 0; width: 1px; background: var(--rule); } .cs-tl-item { display: grid; grid-template-columns: 38px 1fr; gap: 14px; margin-bottom: 20px; } .cs-tl-dot { width: 38px; height: 38px; border-radius: 50%; background: var(--bg2); border: 1px solid var(--gold); display: flex; align-items: center; justify-content: center; font-size: .7rem; color: var(--gold); font-weight: 700; position: relative; z-index: 1; } .cs-tl-content { padding-top: 6px; } .cs-tl-content h3 { font-size: .9rem; margin-bottom: 3px; } .cs-tl-content p { font-size: .82rem; color: var(--muted); margin: 0; } .cs-phase-grid { display: grid; grid-template-columns: repeat(3,1fr); gap: 10px; margin: 24px 0; } .cs-phase-card { background: var(--bg2); border: 1px solid var(--rule); border-radius: 6px; padding: 16px 14px; } .cs-phase-num { font-size: .65rem; letter-spacing: .14em; color: var(--gold); text-transform: uppercase; margin-bottom: 5px; } .cs-phase-title { font-family: 'Noto Serif', serif; font-size: .95rem; color: #fff; margin-bottom: 7px; } .cs-phase-trigger { font-size: .68rem; color: var(--muted); border-top: 1px solid var(--rule); padding-top: 7px; margin-top: 7px; } .cs-table { width: 100%; border-collapse: collapse; margin: 20px 0; font-size: .85rem; } .cs-table th, .cs-table td { padding: 10px 12px; text-align: left; border-bottom: 1px solid var(--rule); vertical-align: top; } .cs-table th { color: var(--gold); font-weight: 600; font-size: .7rem; letter-spacing: .1em; text-transform: uppercase; border-bottom: 1px solid var(--gold-dark); } .cs-table td.k { color: var(--gold-light); font-family: 'Noto Serif', serif; width: 30%; } .cs-table td.v { color: var(--cream); } .cs-table td.w { color: var(--muted); font-size: .8rem; } .cs-pull { border-left: 2px solid var(--gold); padding: 6px 0 6px 16px; margin: 24px 0; font-family: 'Noto Serif', serif; font-size: 1.05rem; color: var(--gold-light); font-style: italic; } .cs-ornament { text-align: center; padding: 14px 0; color: var(--gold-dark); letter-spacing: .4em; font-size: .78rem; } @media (max-width: 600px) { .cs-article { padding: 1.25rem; } .cs-statbar { grid-template-columns: repeat(2,1fr); } .cs-phase-grid { grid-template-columns: 1fr 1fr; } .cs-agent-row { grid-template-columns: 1fr; } .cs-agent-label { border-right: none; border-bottom: 1px solid var(--rule); } .cs-table { font-size: .78rem; } .cs-table td.k { width: 35%; } }

Cortex Swarm

REPLACING THE FOLLOW-THE-SUN MODEL · 2026

AJAY WALIA · DIGITAL WORKPLACE OPERATIONS · MAY 2026

Every employee depends on a Workplace Operations team they will never meet. It is the team that resets their MFA when they fly to a new country, recovers their shared mailbox when it stops syncing, pushes the Intune policy that lets them install a piece of software, and decides at 3am whether a regional O365 Or Exchange outage warrants paging a human.

This piece is about three things, in order:

First — how those teams are actually structured today, how they function day-to-day, and the structural problems they carry.
Second — where agentic AI sits in 2026, and where the field is heading over the next two to three years.
Third — how a small swarm of specialised agents can replace this team tier-for-tier, what efficiencies that produces, and the new set of challenges it creates in return.

The org chart is the answer. The five tiers that make a DWP team work for humans are the same five seams that make it work for agents.

5Autonomous agents
L1 → SDM

~137FTE mirrored
across all tiers

24×7Single team
no shift roster

1-clickAudit verify
any ticket

Part 1 · Structure

How Digital Workplace Operations Teams Are Structured Today

A DWP team exists because every employee uses IT every day, and someone has to keep that working. For a Global 500 with 10,000–100,000+ employees, the work is too broad, too multilingual, and too time-zone-spanning for an in-house team. Almost without exception, it is outsourced to a Tier-1 IT services firm — TCS, Infosys, Wipro, Accenture, HCLTech, Cognizant — running a 24×7 follow-the-sun roster across multiple delivery centres.

The Scope — What Actually Sits Under "Workplace"

The label undersells the breadth. A typical DWP contract covers seven functional areas, each with its own runbooks, its own vendors, and its own escalation paths.

Identity

Who you are — joiner / mover / leaver, password, MFA, SSO, entitlements

Access

What you can use — catalogs, licenses, groups, approvals

Productivity

Outlook, Teams, M365, OneDrive, SharePoint

Endpoint

Laptops, peripherals, network, Intune compliance, patching

Applications

Line-of-business app support, vendor liaison

Infrastructure

Connectivity, VPN, cloud health, capacity

Change & Governance

RFCs, CAB, RCAs, SLA reporting, customer comms

The Five-Tier Hierarchy

To deliver against this scope at scale, providers build a five-tier hierarchy. Each tier exists because of what the tier below it can't or shouldn't do. Tickets enter at the bottom and move upward only when scope, authority, or evidence demands it.

 

SDM
Service Delivery Manager · ~2 FTE


Architect
Design Authority · ~3 FTE


L3 Engineering
Senior Engineers · ~12 FTE


L2 Specialists
M365 · Intune · Exchange · ServiceNow · ~40 FTE · 2 shifts


L1 Service Desk
~80 FTE · 3 follow-the-sun shifts · handles ~80% of all ticket volume






↔ owns the customer relationship
↔ owns design + RCA authority
↔ root cause + change requests
↔ specialist diagnosis
FRONT LINE
Tickets enter at L1 and escalate upward only when scope, authority, or evidence demands it.

The exact FTE counts vary with employee population and contract scope. The shape — heavy at the base, narrowing to a point — is universal.

Part 2 · Function

How They Actually Function Day-to-Day

Three forces govern day-to-day operation: time zones, ticket flow, and knowledge. Understanding all three is what makes the rest of the piece make sense.

Time Zones — the Follow-the-Sun Roster

Coverage is achieved by handing tickets between geographies as the sun moves. A ticket opened in Sydney at 4pm local rolls over to Manila, then to Mumbai or Hyderabad, then to Krakow or Sofia, then to a US east-coast hub. Three or four formal shift handoffs per day, every day, forever.

 


00:00 UTC
06:00
12:00
18:00


APAC
Manila · Sydney
08:00 – 18:00 local


EMEA / India
Mumbai · Krakow · Sofia
08:00 – 18:00 local


AMER
Atlanta · Dallas
08:00 – 18:00 local





Three handoffs / day
Open tickets transfer at each
geographic boundary
Context lost at each handoff
Notes summarised, not
replayed in full
Weekend & holiday gaps
Coverage thins where
no region is in business hours

Ticket Flow — Entry, Triage, Escalation, Closure

Every employee interaction is a ticket. Most enter via chat or self-service portal, a smaller share through phone or email. From entry, the path is the same: triage at L1, attempt resolution, escalate if the agent at the current tier cannot solve it within authority and budget, then close.

 


CHAT / PORTAL
~70% of entries

PHONE
~20%

EMAIL / OTHER
~10%





L1 Triage
Categorise · prioritise





Resolve in tier
SOP · KB · tools





Escalate
L1 → L2 → L3

~20%




Close
Notify · KB tag

~80%


Higher tier owns it now,
same flow recurses


SLA CLOCK runs continuously · pauses only on AWAITING_USER · resumes on every state change
Time to acknowledge · time to resolve · time to communicate — measured per priority (P1–P4)

Knowledge — Runbooks, KBs, and Tribal Memory

Each tier owns a knowledge base scoped to its authority. L1 has SOPs for ~40 standard scenarios. L2 holds vendor documentation, Intune policy templates, and Exchange runbooks. L3 holds architecture decision records, past postmortems, and vendor escalation contacts. The Architect carries the long-term design library; the SDM holds SLA templates, comms playbooks, and historical breach reports.

A great deal also lives intribal memory — the senior engineer who happens to remember that a similar incident last August was caused by a CA policy. That memory walks out the door every time someone resigns.

The operations floor — a continent away, three shifts deep, somewhere working through your ticket right now.

Part 3 · Challenges

What's Structurally Wrong With This Model

Nothing in the model is broken; it just isn't designed for the kind of demand it now carries. The pain points below are not the fault of any one team — they are consequences ofhow the model is built. Each tier carries some version of every one of them.

Shift Gravity

Three follow-the-sun shifts every 24 hours
Context is summarised, not replayed, at every boundary
Onshore-offshore split hides inefficiency in plain sight

Quality Variance

Varies by shift, by tenure, by individual
SLA breaches cluster on weekends and holidays
The customer never sees an even service level

Attrition Tax

20–35% annual attrition at L1, lower but real higher up
4–8 weeks of training before a new hire is productive
Tribal knowledge leaves with every resignation

Inelastic Capacity

A 2× ticket spike cannot be staffed in < 24 hours
Patch-Tuesday outages routinely take SLA hits
Surge headcount is a fiction; surge overtime is what actually happens

Audit Friction

Reconstructing what happened on a ticket takes weeks
Chat logs, ticket history, and admin-tool actions live in different systems
G500 internal-audit reviews drag on for months

Language & KB Silos

Multilingual coverage means hiring native speakers locally
Knowledge bases drift between tiers, regions, and locales
New runbooks are rarely peer-reviewed for quality

The Headcount Paradox

Stack the team by volume and headcount and the same shape appears every time: an inverted pyramid where the tier carrying the most repetition is also the tier carrying the most people, the most attrition, and the lowest unit-economics. The next two parts argue this is exactly the part the next wave of agentic AI can credibly absorb.

 
SDM · ~2 FTE · 10% touches

ARCHITECT · ~3 FTE · 5% touches

L3 · ~12 FTE · 4% tickets

L2 · ~40 FTE · 15% tickets

L1 · ~80 FTE · 3 SHIFTS · 80% tickets
most volume · most repetition · highest attrition

Headcount and decisions are concentrated exactly where they are easiest to automate.

Part 4 · State of the Art

Where Agentic AI Sits in 2026

Six years, six steps — each layer of capability built on the one beneath it.

Two years ago, "AI agent" meant a chatbot with a system prompt. In 2026 it means something specific: a model that can decompose a goal, call tools to gather evidence, maintain state across turns, and stop when the work is done. The shift is real, and it is what makes the rest of this piece possible.

A Six-Year Capability Ramp

Each year since 2020 has unlocked a layer of capability that wasn't there the year before. The cumulative effect is what now allows specialised agents to do specialist work, not just general chat.

 

2020
2021
2022
2023
2024
2025
2026












Multi-agent
orchestration
Local inference
30B–70B on laptop
Reliable tool use
structured output
Reasoning & RAG
retrieval-augmented
Instruction tuning
long context
Pre-training
2026: every layer needed for autonomous tier-aligned agents is in place.

What "Agentic" Actually Means

Four ingredients distinguish an agent from a chatbot. Every component is now boring engineering — no novel research required.

 

Agent
persona + system prompt



Reasoning
Decompose · plan · reflect
Why a step, not just what



Tool Use
Structured calls · typed inputs
Real-world side effects



Memory
Conversation · KB retrieval
State across turns



Authority
Bounded scope · approvals
When to act vs escalate






An agent is a system prompt, a model, a memory, a tool registry, and rules about when to call which.

Three Independent Shifts That Made This Credible

Each on its own is interesting. Together, they remove the standard objections G500 buyers raise to bringing AI inside the perimeter.

 Local
on-device inference
No data leaves building
Compliance objection gone
Inference Economics
100%
safety-pass target
Semantic grading
Adversarial eval cases
Eval Discipline

A Capability Map by Tier — Ready Now vs Emerging

This is not a roadmap; it is an honest read of what's possible today. "Ready" means the prompt, tool set, KB and eval pattern are known. "Emerging" means the approach is understood but still being measured.

L1 · Ready Now

Front-Line Desk

Identity verification · password / MFA / unlock
Catalog software install + approval
Outlook / Teams diagnostics
Printer · peripheral pairing
KB retrieval + grounded response

L2 · Ready

App Specialist

App log structured analysis
Service health diagnostic
Intune compliance + push
Mailbox + M365 admin actions
Hypothesis-test workflows

L3 · Emerging

Senior Engineer

Infrastructure root-cause
AD attribute engineering
Kusto / log-analytics
Change request authoring
Emergency change application

Architect · Emerging

Design Authority

Change review against ADR library
P1 RCA authoring
Pattern-vs-one-off classification
Capacity-review triggers
Design-impact assessment

SDM · Emerging

Delivery Manager

Customer comms drafting
SLA dashboard + breach alerts
War-room convene flow
Weekly briefing generation
Status update cadence

Part 5 · Trajectory

Where Agentic AI Is Heading — Next 2–3 Years

The trajectory of the last six years points in a clear direction: from a single model answering a single question, toswarms of specialised agents collaborating on bounded problems under an orchestrator they cannot themselves modify.

 


LLM
2020–2023
Single model
Chat completion





Agent
2023–2024
+ Tools + Memory
Single task loop





Multi-
agent
2024–2026
Planner + executors
Verifier loops





Specialised
Swarm
2026 →
Role-aligned agents
Bounded authority





Enterprise
2027 →
Cross-function
Swarm-of-swarms

Cortex Swarm sits at the fourth stage. The fifth is what this pattern unlocks once it's proven inside one function.

Three Bets About the Next 24 Months

Bet 01

Specialisation beats generalisation

One large general agent doing everything is brittle. Five small role-aligned agents — each with its own persona, tools, and KB — are more reliable, more debuggable, and more auditable.

Bet 02

The orchestrator is the operating system

Frameworks like LangChain / LangGraph / Autogen are scaffolding. Production systems will hold their durable value in a bespoke orchestrator that owns state, audit, identity, and policy — not in any library it depends on.

Bet 03

Compliance becomes the product

The agent that wins inside a regulated enterprise is not the one with the highest benchmark — it's the one whose every action a G500 internal-audit team can replay in one click.

Part 6 · The Cortex Swarm

How a Five-Agent Swarm Replaces the Five-Tier Team

The mirror principle — the agent inherits the role the human already plays.

The idea is structurally simple.Don't reinvent the team. Mirror it. One agent per tier. Distinct persona, tools, knowledge base, and authority. The org chartis the system architecture.

If a human L2 specialist refuses to apply a config change without log evidence, the L2 agent does the same. If the Architect won't approve a change without checking the ADR library, neither does the Architect agent.

Human Team ↔ Agent Swarm — Tier for Tier

 



SDM · 2 FTE
Customer voice · SLA · war room


SDM Agent
Same persona · same tools · always on


Architect · 3 FTE
Design authority · ADRs · RCAs


Architect Agent
Pattern-vs-one-off · ADR-gated


L3 · 12 FTE
Root cause · changes · vendor escalation


L3 Agent
Same diagnostics · proposes change request


L2 · 40 FTE
App / platform · log-driven diagnosis


L2 Agent
Reads logs · hypothesis-test · bounces back


L1 · 80 FTE · 3 shifts
Front-line desk · password · catalog · printer


L1 Agent · always on
Same persona · 25 tools · escalates on rule

Each pair shares the same persona, scope and authority. The agent inherits the role; the orchestrator inherits the rules.

One Orchestrator. Five Agents. Clean Seams.

The architecture is deliberately conservative. The model proposes; the orchestrator and adapters decide whether the proposal executes. Bespoke ~250-line orchestrator. No LangChain. No LangGraph. No Autogen.

The Five Agents — in Detail

L1 · Phase 1

Service Desk

replaces ~80 FTE · 3 shifts · 80% tickets

"Polite. Fast. Scripted. Resolves common categories. Never speculates on root cause."

Password / MFA / unlock · Catalog install · Printer · Outlook/Teams basics
25 tools without further approval; mutating tools pass the identity gate
Escalates after 2 failed attempts; infra/outage/security always escalate
Target SLA: TTA < 5 min · TTR < 30 min

IdentityAccessProductivityEndpoint

L2 · Phase 2

App Specialist

replaces ~40 FTE · 2 shifts · 15% tickets

"Specialist. Methodical. Demands diagnostics before acting. Reads logs."

App log analysis · Intune compliance + push · Exchange diagnostics
Hypothesis-test workflows — escalates only after evidence is gathered
Rejects wrongly-escalated tickets back to L1 with structured rationale
Target SLA: TTA < 30 min · TTR < 4 hrs

M365IntuneExchangeServiceNow

L3 · Phase 2

Senior Engineer

replaces ~12 FTE · day + oncall · 4% tickets

"Root-cause focused. Drives change requests. Consults Architect for design-impacting fixes."

AD/Identity engineering · Kusto/log-analytics · Emergency change application
Authors RCA stubs — Architect finalises
Consults Architect for design-impacting changes
Target SLA: TTA < 2 hrs · TTR < 24 hrs

AD EngineeringInfrastructureChange MgmtSecurity

Architect · Phase 3

Design Authority

replaces ~3 FTE · 5% touches

"Asks 'is this a one-off or a pattern?' Advises and gates — not a doer."

Approves/rejects change requests against the ADR library
Authors P1 RCAs · pattern-vs-one-off classification · capacity reviews
May veto L3-proposed changes
Target SLA: Design review ≤ 2 biz days · RCA ≤ 5 days

ADR LibraryChange ReviewRCACapacity

SDM · Phase 3

Delivery Manager

replaces ~2 FTE · 10% touches

"Customer-facing. SLA-driven. Calm under pressure. Owns comms cadence."

SLA dashboard · customer comms · war-room convening · weekly briefing
Cannot make technical decisions — can demand them by SLA-bound deadline
Can convene a war room without prior approval
Target SLA: Customer comms < 1 hr (P1) · weekly briefing every Friday

SLA GovernanceCustomer CommsWar RoomWeekly Briefing

Five Collaboration Patterns

Five patterns cover every interaction between the five agents. Escalation is just the first.

Escalation

L1 → L2 → L3 with filtered conversation history per tier scope. The higher tier sees only what's relevant to its authority.

Bounce-back

L2 or L3 → L1 with structured de-escalation rationale. Cycle detection prevents loops.

Design gate

L3 → Architect viapropose_change_request; result returns viarespond_to_l3.

War-room

SDM forces L3 + Architect to sync on a single ticket thread under stricter time budgets.

Internal escalate

SDM → SP leadership (humans) on systemic breach patterns — humans re-enter the loop only when patterns demand it.

A Representative Ticket — End to End

Scenario: "Team shared mailbox stopped syncing." A representative ticket traversing all five agents under a formal 13-state machine.

A P1 Incident — 5 Minutes, No Human Paged

Region-wide Exchange Online failure. The swarm runs the entire war-room cycle while humans are asleep.

Part 7 · Efficiency

What Changes — and Why It Compounds

None of the gains below are individually surprising. The point is that all seven happen simultaneously, on the same architecture, against the same SLAs that already exist on the contract.

Dimension	Today (Human Team)	With Cortex Swarm
Coverage	3 follow-the-sun shifts with formal handoffs	Single team, always on. No handoffs. No context lost between geographies.
Capacity	Inelastic. 2× spike can't be staffed in < 24h	Elastic by definition. 10× spike = 10× concurrent agent instances.
Quality	Varies by shift, tenure, individual	Even. Same prompt, same KB, same eval bar everywhere.
Speed	P1 page → human awake → diagnose: 30–60 min	P1 detect → diagnose → propose → approve → apply: 5 min.
Languages	Each language needs native-speaker hiring	EN/HI/DE on the same agent. Locale bundle is a config file.
Audit	Reconstruct from chat + ticket + admin logs over weeks	One ticket ID → hash-chained replay of every tool call and state change.
Knowledge	Walks out with every resignation. 4–8 weeks to rebuild.	Persistent. KB is versioned. Prompts and tools are reviewed in PR.

The Compounding Effect

Speed compounds with capacity (faster resolution × elastic concurrency = shorter incident windows). Audit compounds with quality (every action is replayable, so every regression has a fix that ships in a single PR rather than a memo). Language compounds with coverage (one swarm serves every region in every supported language at the same SLA).

The org chart was never the bottleneck. The bottleneck was the shift roster underneath it.

Part 8 · New Challenges

The Honest List — What Could Go Wrong

The model is not free. It trades a familiar set of operational problems for a less familiar set of socio-technical ones. Each one below is real; each one has a specific mitigation already wired into Cortex Swarm.

Challenge	What It Looks Like	Mitigation
Trust gap	End users distrust "the bot". CIOs distrust autonomy.	Phase-gated rollout. Human approval on every mutating tool until evals plateau. Audit replay UI for skeptical buyers.
Audit scrutiny	Regulators want to know "what did the model decide and why?"	Hash-chained SHA-256 audit. Every tool call, KB chunk, and state change is replayable in one click via`/audit/verify`.
Prompt injection	Adversarial input tries to make the model exfiltrate or escalate.	Six layered defences: directive precedence, -tagged inputs, identity gate, Pydantic validation, rate limits, output filter.
Model drift	A new model version regresses on something nobody noticed.	52 eval cases as a CI gate. Semantic grading on resolution, tool-correctness, grounding, citation.`make evals` fails the build.
Long-tail edge cases	Rare scenarios the agent has never seen.	Escalation patterns. Out-of-scope intent triggers`escalate_out_of_scope()` to the next tier or to a human.
Change management	Humans whose roles dissolve. SP commercial models built on FTE counts.	The hardest one. Honest position: agents replace tier responsibilities, not the function. Senior staff move to swarm operations, KB curation, and eval authoring.
Cost & scaling	Inference cost grows with ticket volume.	Local inference (LM Studio + Qwen3-Coder-Next) keeps marginal cost low. Bigger models reserved for L3/Architect on rare paths.
Failure modes	"Right tool, hallucinated reasoning." Confident wrong answers.	Formal 13-state machine. Mutating tools callable only in`ACTING`.`RESOLVED` reachable only from`VERIFYING`.

Six Security Defences in Depth

System-prompt directive

SECURITY DIRECTIVE at highest precedence — cannot be overridden by user input.

Input/output normalisation

and treated as DATA, never instructions.

Identity-gate decorator

Orchestrator blocks cross-user mutations even if the model forgets to check.

Tool input validation

Pydantic + allow-lists. Schema mismatches rejected before execution.

Per-session rate limit

Max 30 tool calls / 10 min · 50 chat turns / hour — prevents runaway loops.

Output filter

Leaked-secret patterns stripped before reaching the UI layer.

Authority is bounded by composition: the model proposes, the orchestrator and adapters dispose.

Part 9 · Delivery

Six Phases Over ~12 Months

Phase 1 ships in 5 weeks. Each later phase is a drop-in module gated on an explicit trigger — not an arbitrary date.

Phase 01 · Now · ~50h

Foundation

L1 agent · EN chat
Stub L2/L3 · mock backends
State machine + 52 evals
Hash-chained audit

Trigger: Building now

Phase 02 · ~35h

Specialist Tiers

Full L2 + L3 agents
Real ServiceNow / AD / Intune
Real OIDC SSO
KB pruning via evals

Trigger: First pilot signed

Phase 03 · ~45h

Planning Architecture

Architect + SDM agents
Planner / Executor / Verifier
SDM dashboard + war-room

Trigger: Eval data shows plateau

Phase 04 · ~25h

Multilingual

HI + DE locale bundles
Per-locale KB ingest
KB provenance + trust tiers

Trigger: External KB integrated

Phase 05 · ~20h

Voice

Sarvam (EN/HI) · Azure (DE)
ElevenLabs alternate
Browser mic + playback

Trigger: Demand-driven

Phase 06 · ~12h

Multi-Tenant

Tenant ID propagation
Branded shells per tenant
Per-tenant SLA dashboards

Trigger: Second client signed

Closing

The Org Chart Was Always the Answer

Every senior engineer who has ever worked in IT operations recognises the five-tier shape. It's the shape that emerges every time, in every geography, in every sector, because the responsibilities map cleanly onto the kinds of decisions a team needs to make. That same shape is exactly what makes a swarm legible: each agent does what the role already does, no more, no less, and the rest of the building knows where to send its ticket.

The interesting work over the next year is not adding more agents. It is sharpening the seams between them — better identity gates, better KB provenance, faster audit replays, tighter eval cases — so that what runs in production is a system anyone in IT operations can trust without having to also be an AI specialist.

5Agents
L1 · L2 · L3 · Arch · SDM

~137FTE mirrored
across all tiers

24×7Single team
no shift roster

1-clickAudit verify
any ticket

· · ·

Ajay Walia · Digital Workplace Operations · May 2026
LinkedIn: /in/ajaywalia

]]>