Product PR #12
Refund timing changed to 5–7 business days
AI agents · GitHub APIs · Documentation automation
The product changed. The help docs should know.
KB DocBot analyses merged product pull requests, checks the customer help documentation and prepares an update when the product and the docs no longer agree.
It does the repetitive investigation and drafting work. A person still owns what gets published.
Documentation agent
Checking recent product changes
Help documentation checked
Existing refund guidance is out of date
Documentation update proposed
Preparing a GitHub pull request for review
Decision
create_pr
Human review
Required
Connected repositories: product, help documentation and orchestration app
Coherent documentation PR created from multiple product changes
AI-generated changes published without a person reviewing them
The problem
Documentation drift usually starts quietly.
A product changes, the code ships and the help content continues describing yesterday’s behaviour.
The information needed to catch that gap is scattered. Engineers understand the code change. Writers understand the documentation. Support teams know where customers are getting confused. Connecting all three is often a manual process.
I wanted to explore whether an AI agent could make that connection without giving it unchecked control over customer-facing content.
The workflow
From merged code to a reviewable documentation change.
Each product PR is analysed separately, but the final output is one coherent documentation update.
Fetch
Retrieve merged, unprocessed product pull requests from GitHub.
Titles · bodies · changed files · patchesInspect
Process each PR separately so the agent can reason about one product change at a time.
Oldest merge firstCompare
Read the current Markdown documentation and locate content describing the changed behaviour.
Product behaviour ↔ help contentDecide
Return either a no-change decision or a structured proposal supported by evidence.
Typed agent outputCombine
Carry earlier documentation updates forward while later product behaviour takes precedence.
Working documentation snapshotReview
Create the branch, commit and documentation pull request for a person to inspect.
No automatic publishing
Product repo
SMTH Payments
Orchestration
Next.js + DocBot agent
Documentation repo
Help content PR
Final decision
Human reviewer
The interesting engineering problem
What happens when several PRs change the same thing?
My first approach generated a complete documentation file for every product PR and tried to combine the results afterwards.
That was unreliable. Two proposed files could look conflicting even when the underlying product changes were unrelated.
The solution was to process the PRs in chronological merge order and maintain a working copy of the help documentation.
- Different behaviour changed? Preserve both updates.
- Same behaviour changed again? The latest merged PR wins.
- Unrelated content on the same page? Leave it intact.
Refunds change from immediate to 3–5 days.
Docs: 3–5 days
Payouts change from once daily to twice daily.
Docs: refunds 3–5 days + payouts twice daily
Refunds change again, this time to 5–7 days.
Final docs: refunds 5–7 days + payouts twice daily
Human-in-the-loop by design
The agent can draft. It cannot quietly decide what customers are told.
Every change is reviewable
DocBot creates a normal GitHub pull request containing the proposed edits, the evidence behind them and links to the source product PRs.
A reviewer can inspect the diff, edit the wording, approve it, request changes or reject it.
Good robot / Bad robot
Each documentation PR asks the reviewer to react to the description:
👍
Good robot
Accurate and useful
👎
Bad robot
Incorrect or unnecessary
Written comments and human edits provide richer evidence than a rating alone.
Deliberately lightweight
GitHub is the prototype’s memory and audit trail.
Labels mark successfully processed product PRs so future runs do not repeat the work.
The generated PR already stores the proposal, evidence, reviewer reactions, comments, human edits and final outcome.
That gives the prototype meaningful persistence without adding a database before it actually needs one.
Thinking beyond the prototype
GitHub should remain the evidence. It should not remain the entire production data layer.
At company scale, I would keep GitHub as the source of truth and add webhooks plus a structured database for efficient retrieval, analytics and organisation-specific learning.
01
Reviewer feedback
Reactions, comments, edits and PR outcome
02
GitHub webhook
Send structured activity to the application
03
Feedback store
Searchable decisions, outcomes and examples
04
Approved guidance
Human-reviewed improvements for future runs
One important boundary:
I would not allow the agent to rewrite its own instructions automatically. An evaluation agent could propose guidance changes, but a person would review them before they affected future decisions.
Technology
More than a prompt attached to GitHub.
The project required application architecture, authentication, API integration, structured agent outputs, repository state and failure handling.
Next.js
React
TypeScript
Node.js
OpenAI Agents SDK
Structured outputs
GitHub REST API
GitHub GraphQL API
GitHub App authentication
Pull requests
Branches and commits
Markdown
API debugging
Human-in-the-loop AI
Feedback architecture
What this project demonstrates
The kind of work I enjoy sits between systems and people.
Building DocBot meant thinking about code, APIs and agents, but also about customer impact, trust, auditability and where human judgement belongs.
It brings together the parts of technical work I enjoy most: investigating how a system behaves, designing a practical workflow and turning complicated changes into information people can actually use.
Explore the repositoryNext project