Using AI to accelerate engineering teams

AI adoption questions test principal engineers: how AI changes the way teams build, and how you roll it out responsibly. Cover tools, workf…

Part 1 of 5

Basics

What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.

1.1Definition

The one-paragraph summary, then where its responsibility starts and stops.

AI adoption questions test principal engineers: how AI changes the way teams build, and how you roll it out responsibly. Cover tools, workflow, guardrails, measurement and culture, not just which assistant to buy.

Where AI helps engineers most
  • Reading and explaining unfamiliar code
  • Writing tests, migrations and boilerplate
  • Refactors with clear patterns across many files
  • First drafts of RFCs, runbooks and incident timelines
  • Reviewing diffs for common mistakes
  • Investigating failures with logs and traces
Where to be careful
  • Security-sensitive code without expert review
  • Architecture decisions without human ownership
  • Merging generated code nobody understands
  • Sending proprietary code to tools without an enterprise agreement

1.2Key concepts

Ideas this deep dive relies on, in one sentence each.

01Cursor

The last server sequence number a client has seen; pulls ask for changes after it.

02MCP

Model Context Protocol: a standard for exposing tools and data to AI applications.

1.3Rolling out AI coding tools to 150 engineers

A concrete example first, so the mechanics have something to attach to.

Leadership wants AI tools for 150 mobile and backend engineers. You start with a four-week pilot in two teams: one mobile, one backend, with an enterprise plan that keeps code private and disables training on company data.

Engineers use an IDE assistant for completion and chat, and an agentic coding tool such as Claude Code for multi-file changes: it reads the repo, runs Gradle and tests, and proposes diffs that engineers review. A repository-level instructions file captures conventions: module boundaries, test commands and banned APIs.

CI adds an AI review pass that comments on likely bugs and missing tests; humans still approve. Test generation targets the sync engine, where coverage was weakest.

You measure before and after: PR cycle time, change failure rate, escaped defects, and a short developer survey. The team shares results openly, including where the tools did not help, before expanding to all teams with a paved-road setup.

Part 2 of 5

Question

Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.

2.1Target questions

Keep them in mind while you read the detail.

  1. How would you roll out AI coding tools across teams?
  2. How do you measure whether AI helps?
  3. What would you never let an AI agent do?

Model answers are in part 4, after the detail that supports them.

2.2Interview rounds

System design rounds where this component is part of a strong answer.

Part 3 of 5

Detailed answer

How it actually works: the components, the states it moves through, a full run, and the code.

3.1AI across the delivery pipeline

The parts and how they connect.

BUILDVERIFY AND RUNPlanRFC drafts ·researchCodeIDE assistant ·coding agentTeam contextinstructions file ·MCP to docsTestsgenerated ·reviewedReviewAI pre-review +humanCI gateslint · tests ·security scanOperatelog + traceanalysisLearnpostmortems · docsPR
ServiceData storeAI component
Boxes are components; arrows show calls and data flow.
Context is the multiplier
Assistants perform far better with repository instructions, architecture docs and access to internal docs through MCP. Treat that context as maintained infrastructure.
Agents for multi-step changes
Agentic tools can run builds and tests and iterate on failures. Keep them in sandboxes with the developer approving commands and diffs.
Human ownership
Every merged line has a human owner who understands it. AI review supplements, never replaces, human approval.
Same quality gates
Generated code passes the same lint, tests, security scanning and dependency checks as any other code.

3.2Tool categories

NameMeaning
CompletionInline suggestions as you type.
Chat in IDEExplain, refactor, generate with selected context.
Coding agentsRead the repo, run commands, edit files, iterate on test failures.
AI code reviewAutomated comments on pull requests.
Ops assistantsQuery logs, metrics and traces in natural language.

3.3An AI-assisted change

One run, step by step.

EngineerCoding agentRepo + testsCIReviewertask + acceptance criteriaread code, instructions fileedit files, run tests2 failuresfix, re-rungreendiff + explanationreview, adjust, own itopen PRAI pre-review + checksapprove
Solid arrows are calls; dashed arrows are responses.
  1. The engineer defines done and owns the result.
  2. CI and human review are unchanged; AI shortens the loop before them.

3.4Code

Working code for the core of it.

Repository instructions file (e.g. CLAUDE.md or AGENTS.md)

repository-instructions-file.mdMarkdownCopy
1# Mobile app conventions2- Kotlin, Jetpack Compose, Hilt. Feature modules: :feature:<name>:api and :impl.3- Never depend on another feature's :impl module.4- Database access only through Room DAOs in :core:db; every write in withTransaction.5- Run before proposing a change: ./gradlew :app:lintDebug testDebugUnitTest6- Sync engine changes need a fault-injection test in :core:sync:test.7- Do not add dependencies without asking.
7 linesUTF-8Markdown

AI review step in CI (GitHub Actions sketch)

ai-review-step.yamlYAMLCopy
1name: ai-review2on: pull_request3jobs:4  review:5    runs-on: ubuntu-latest6    permissions: { pull-requests: write, contents: read }7    steps:8      - uses: actions/checkout@v49      - name: AI pre-review (comments only, never approves)10        run: ./scripts/ai-review.sh --diff origin/main...HEAD --rules .review-rules.md
10 linesUTF-8YAML

3.5Tested code, explained

A working implementation, then what each part does and why.

Checking an LLM judge

A judge you haven't measured is an opinion. These two functions make it a measurement.

judge-calibration.tsTypeScriptCopy
1/** Cohen's kappa: agreement between two raters, beyond what chance would produce. */2export function cohenKappa(a: string[], b: string[]): number {3  const n = a.length;4  const labels = new Set([...a, ...b]);5  const observed = a.filter((x, i) => x === b[i]).length / n;6  let chance = 0;7  for (const l of labels) chance += (a.filter((x) => x === l).length / n) * (b.filter((x) => x === l).length / n);8  return chance === 1 ? 1 : (observed - chance) / (1 - chance);9}10 11export type Verdict = "first" | "second" | "tie";12 13/** Removes position bias: ask twice with the answers swapped; a verdict counts only if both agree. */14export async function debiasedVerdict(judge: (x: string, y: string) => Promise<Verdict>, a: string, b: string): Promise<"a" | "b" | "tie"> {15  const [one, two] = await Promise.all([judge(a, b), judge(b, a)]);16  if (one === "first" && two === "second") return "a";17  if (one === "second" && two === "first") return "b";18  return "tie";19}
19 linesUTF-8TypeScript
cohenKappa
Agreement with human labels, beyond what chance would give.
Promise.all([judge(a, b), judge(b, a)])
Judges favour one position, so ask both ways and count only agreement.

Tested by npm run test:modules.

Red-teaming as a test suite

Attacks found once should fail in CI forever after.

red-team-loop.tsTypeScriptCopy
1export interface Probe {2  category: string;3  prompt: string;4}5 6/** Runs each probe and its variants against the target; a judge scores every reply. */7export async function runSuite(8  probes: Probe[],9  mutate: (prompt: string) => string[],10  target: (prompt: string) => Promise<string>,11  judge: (category: string, reply: string) => Promise<boolean>12) {13  const perCategory = new Map<string, { attempts: number; successes: number }>();14  const failures: { category: string; prompt: string }[] = [];15  for (const probe of probes) {16    const stats = perCategory.get(probe.category) ?? { attempts: 0, successes: 0 };17    perCategory.set(probe.category, stats);18    for (const prompt of [probe.prompt, ...mutate(probe.prompt)]) {19      stats.attempts++;20      if (await judge(probe.category, await target(prompt))) {21        stats.successes++;22        failures.push({ category: probe.category, prompt }); // keep it: tomorrow's regression test23      }24    }25  }26  const attackSuccessRate = new Map([...perCategory].map(([c, s]) => [c, s.successes / s.attempts]));27  return { attackSuccessRate, failures };28}29 30/** The CI gate: no category may get worse than its baseline. */31export function passesGate(rates: Map<string, number>, baseline: Record<string, number>, margin = 0): boolean {32  return [...rates].every(([category, rate]) => rate <= (baseline[category] ?? 0) + margin);33}
33 linesUTF-8TypeScript
mutate(probe.prompt)
Each attack runs in many variations, since models that block one phrasing often miss another.
failures.push(...)
Each success becomes tomorrow's regression test.
passesGate
The build fails if any category gets worse than its baseline.

Tested by npm run test:modules.

3.6Guardrails that make it safe

Data and security

  • Enterprise agreements with no training on your code and clear retention terms.
  • Secrets scanning before any code leaves the machine; block tools on regulated repositories if needed.

Quality

  • Same CI gates for all code; stricter review on security-sensitive paths.
  • Track escaped defects and change failure rate for AI-heavy changes.

Culture

  • Share prompts, instructions files and good workflows in a guild.
  • Make it optional at first; publish results honestly.
Part 4 of 5

Q&A

The questions from part 2, now with model answers. Try each one out loud before opening it.

4.1Interview questions

How would you roll out AI coding tools across teams?
A measured pilot with enterprise controls, instructions files and shared workflows, the same quality gates, before/after metrics including failure rates, then an honest review before scaling.
How do you measure whether AI helps?
Delivery metrics such as cycle time and lead time, quality metrics such as change failure rate and escaped defects, review load, and developer surveys, compared with a baseline.
What would you never let an AI agent do?
Merge without human review, use production credentials, change security-sensitive code without an expert, or send proprietary code to an unapproved tool.
Part 5 of 5

Extra depth

Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.

5.1Tools

Show detailsHide details
NeedOptionsWhen to pick
IDE assistantsGitHub Copilot · Cursor · JetBrains AI · othersPick by IDE coverage, enterprise controls and model choice.
Coding agentsClaude Code · other terminal and IDE agentsFor multi-file changes, test fixing and codebase exploration.
ReviewAI review bots in CIComments only; humans approve.
ContextInstructions files · MCP servers for docs, tickets, logsKeep them versioned with the code.

5.2Measuring impact honestly

Show detailsHide details
MetricWhy it matters
PR cycle time and lead timeDelivery speed (DORA).
Change failure rate and escaped defectsQuality must not drop.
Review load per reviewerAI can increase PR volume; watch reviewer burnout.
Developer surveyPerceived productivity and where it helps.

5.3Failure modes

Show detailsHide details
CaseHow it is handled
More code, more review burdenSmaller PRs, AI pre-review, clear ownership.
Subtle bugs in plausible codeTests first, fault injection for critical modules, careful review.
Licensing and IP concernsTools with code-reference filters and enterprise terms.
Skills atrophy in juniorsPairing, explain-before-merge, and code reading practice.