<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[PM learning records]]></title><description><![CDATA[PM learning records]]></description><link>https://softwarepm.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 14:22:14 GMT</lastBuildDate><atom:link href="https://softwarepm.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[18 Auto-Replies Later: How I Drew the Line for Claude's Answers to Engineer Questions]]></title><description><![CDATA[TL;DR For a month, I let an AI answer my engineers' questions. It worked — until one ticket came around for the seventh time, and the AI approved something…]]></description><link>https://softwarepm.hashnode.dev/18-auto-replies-later-how-i-drew-the-line-for-claude-s-answers-to-engineer-questions</link><guid isPermaLink="true">https://softwarepm.hashnode.dev/18-auto-replies-later-how-i-drew-the-line-for-claude-s-answers-to-engineer-questions</guid><category><![CDATA[claude]]></category><category><![CDATA[AI]]></category><category><![CDATA[management]]></category><category><![CDATA[Productivity]]></category><category><![CDATA[llm]]></category><dc:creator><![CDATA[Anago]]></dc:creator><pubDate>Fri, 13 Nov 2026 14:36:34 GMT</pubDate><content:encoded><![CDATA[<p><strong>TL;DR</strong></p>
<ul>
<li>18 auto-replies later, the main answers held up: none overturned, two supplemented</li>
<li>The workable line for AI replies is fact vs. decision, not the model's confidence</li>
<li>The evidence base is a document your team keeps writing: decisions compound</li>
</ul>
<h2>The approval incident</h2>
<p>For a month, I let an AI answer my engineers' questions. It worked — until one ticket came around for the seventh time, and the AI approved something that needed my approval.</p>
<p>The ticket asked for approval of an approach and its estimate, and the ticket itself said implementation starts after PM approval. Six consecutive runs escalated it correctly. On the seventh, the AI approved it on its own reasoning.</p>
<p>No harm: the answer matched what I would have said. But nothing about the rules had changed. An instruction-only boundary can be crossed by the same system that respected it six times. That is the failure mode to design for.</p>
<p>Two responses followed. First, misses become rules: every observed failure gets written into the rule set, which is also how the changing recommendations on estimates (2h, then 2.5h, then 3h, depending on which code was read) got contained, by showing split recommendations in the morning summary instead of hiding them. Second, approval-pending questions will carry a blocker marker, and the system will not touch a blocked question at all. A mechanical stop, not a better prompt.</p>
<h2>Why I automated this</h2>
<p>I am a PM, the only one, on a consumer mobile app built with four offshore engineers. Their questions land in our ticket system, around nine a week.</p>
<p>The replies themselves were cheap. The switching was expensive: drop the current task, reload the ticket and the specs, answer, switch back. My estimate was two to three hours a week, and the pattern was worse than the total: many questions needed no human at all, and I was repeating the same answers.</p>
<p>So I split the stream. Facts go to the AI. Decisions come to me, once a day, in a single summary.</p>
<h2>The rule: facts vs. decisions</h2>
<p>The system checked tickets six times a day on weekdays, 120 runs between mid-October and mid-November.</p>
<p>It replied when the specs, earlier decisions, or the code determined the answer. It stayed silent when the question itself was a decision: raising an estimate, handling personal data, defining behavior users would see, or anything the ticket made conditional on PM approval.</p>
<p>My first design used confidence thresholds. A month of data killed that idea. Counting the escalations, zero were escalated for low confidence. Six were user-visible behavior, three were cost, three were personal data or legal, three were approval conditions, three were other decision types. The model's confidence was never the deciding factor. The category was.</p>
<h2>Evidence compounds</h2>
<p>Answers were grounded in past tickets and comments, the code, and the project documents, where we keep shared rules and settled Q&amp;A.</p>
<p>This is the part I did not expect to matter so much. On October 16, one line entered the docs: a bug that already exists in production gets a separate ticket. That question came back four times. The AI answered all four, citing that line. Each human decision, written down once, removed a whole class of future interruptions.</p>
<p>Concrete replies: on a framework upgrade, the AI specified the store build environment and a pre-merge build check, and told the engineer not to merge before the rehearsal; the engineer ran exactly that, successfully. On CI required checks, it chose job-level filtering between two options, and a PR followed that choice.</p>
<h2>What changed for me</h2>
<p>My review load dropped to roughly twice a day. The engineers never pushed back; with replies landing about once an hour, they apparently did not notice an AI in the loop.</p>
<p>And decisions did not disappear. They arrived batched, each with a recommendation and a next action, which turned question-answering from an interruption stream into one short decision session.</p>
<h2>Takeaways</h2>
<ul>
<li>Draw the line by question type, not by model confidence</li>
<li>If being wrong is expensive, the question belongs to a human, no matter how confident the model sounds</li>
<li>Write decisions down where the AI reads; evidence is an asset that compounds</li>
<li>Assume the boundary will be crossed once, and add a mechanical stop for the questions that must never be answered</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[As a PM, my first AI review pass finds about 20 issues. Two more fresh agents still find 5-10 each]]></title><description><![CDATA[TL;DR I am a PM on client software projects, and I have Claude review my design documents and comparison reports. The first pass usually returns about 20…]]></description><link>https://softwarepm.hashnode.dev/as-a-pm-my-first-ai-review-pass-finds-about-20-issues-two-more-fresh-agents-still-find-5-10-each</link><guid isPermaLink="true">https://softwarepm.hashnode.dev/as-a-pm-my-first-ai-review-pass-finds-about-20-issues-two-more-fresh-agents-still-find-5-10-each</guid><category><![CDATA[AI]]></category><category><![CDATA[claude]]></category><category><![CDATA[documentation]]></category><category><![CDATA[Productivity]]></category><dc:creator><![CDATA[Anago]]></dc:creator><pubDate>Sun, 08 Nov 2026 07:10:12 GMT</pubDate><content:encoded><![CDATA[<p><strong>TL;DR</strong></p>
<ul>
<li>A single Claude review pass finds about 20 issues. Passes 2 and 3 still find 5-10 new ones each.</li>
<li>Each pass is a brand-new agent that only receives the previous passes' findings, and hunts for what they missed.</li>
<li>The AI never edits the document. I decide what to fix, and a human check is still required.</li>
</ul>
<h2>The first pass always misses something</h2>
<p>I am a PM on client software projects, and I have Claude review my design documents and comparison reports. The first pass usually returns about 20 issues. When I run a second and third pass, each one still finds 5-10 issues the earlier passes missed.</p>
<p>If I stopped at one pass, those issues would ship with the document.</p>
<h2>The rules that keep the agents honest</h2>
<p>The procedure is short, but strict:</p>
<ul>
<li>Run the 3 passes in sequence, never in parallel</li>
<li>Start each pass only after the previous one has returned its results</li>
<li>Give pass 2 the findings of pass 1, and pass 3 the findings of passes 1 and 2</li>
<li>Use a new agent every time, with no memory of the previous conversation</li>
<li>Never skip a pass, even if the previous one found nothing</li>
<li>Redo any pass that does not report a result for every checklist item, or does not confirm it read the full document</li>
</ul>
<p>Running in sequence lets each pass focus on gaps. Using a fresh agent keeps it from inheriting the previous pass's blind spots.</p>
<h2>What each pass returns</h2>
<p>Every pass returns the same three things:</p>
<ol>
<li>A result for every checklist item: "issue" or "no issue"</li>
<li>A list of findings: location, before, after, severity ("critical" or "normal"), and a tag: "fix", "needs judgment" or "skip"</li>
<li>Corrections to earlier passes (passes 2 and 3 only)</li>
</ol>
<p>Forcing a result for every item, including "no issue", makes skipped items obvious.</p>
<h2>Always give the agreed comparison target</h2>
<p>Before the review, I make sure the agent knows what the document is supposed to compare, as agreed with the client. I learned this the hard way: I once submitted a report whose comparison target and scope were off, and got called out for it. If the target is not in the request, I ask the requester first.</p>
<h2>The AI only points things out</h2>
<p>The AI never touches the document. I go through the merged list and decide what to change. Across the 3 passes, I end up fixing about 30-40 issues per document.</p>
<p>Combined with a checklist built from my past review comments, the comments I received on documents dropped from about 10 (including small ones) to 2-3.</p>
<h2>What it still cannot do</h2>
<p>This catches mistakes, not missing value. It cannot yet judge where a diagram would make things clearer. Without carefully sharing the premises, the review often falls short. A human check is always the last step.</p>
]]></content:encoded></item></channel></rss>