FIN Analysis Process

Automated process

The fin_analysis GitHub repo automates the Fin analysis process. For a high-level overview with a flow diagram, see the explainer doc.

Every Monday morning (06:00 UTC), a GitHub Actions job:

1. Finds last week's Fin conversations that Intercom marks as closed (meaning the conversation has ended, not necessarily that it resolved the customer's issue) with a CX Score of 1 to 3 (Intercom's CX Score isn't searchable server-side, so the job searches all closed Fin conversations in the window and filters by score client-side).
2. Analyzes each transcript with Claude (running on a Claude subscription via headless Claude Code). The analysis is deliberately sparing: most low scores are customer frustration, misreadings, or off-topic questions, not docs gaps.
3. For real gaps, files a JIRA ticket in DOC labeled FIN:
   • Guidance gap (a rule Fin should operate under): summary prefixed "Guidance:", assigned to the docs lead.
   • Content gap (missing or unclear documentation): unassigned for triage, filed as a Story with Doc Type "Content improvements", routed to the owning docs team (chosen by the analysis), and added to the docs board's next sprint.
   • Every ticket, content and guidance, includes the conversation ID, the Intercom link, a link to the Fin AI dashboard, the concrete improvement, and how it would have helped that specific conversation.
4. Deduplicates: a ticket is never re-filed for the same conversation, and a new ticket that matches an existing open FIN ticket becomes a comment on it instead.
5. Writes a weekly themes report, committed to the repo, and posts a summary to Slack.

Each run opens a pull request with the week's report, assigned to the docs lead for review. See the repo's README for setup, configuration, and how to trigger a manual or dry-run analysis.

Reviewing auto-created tickets

Every ticket that Fin analysis creates includes this note: "This ticket was automatically created by Fin analysis."

Each ticket carries these labels:

• FIN: added to every ticket the automation creates.
• FIN-AUTO-CONTENT or FIN-AUTO-GUIDANCE: identifies the ticket as a content gap or a guidance gap.
• fin-conv-<conversation ID>: a per-conversation label the automation uses to avoid re-filing a ticket for the same conversation. Don't remove it.

When you're assigned one of these tickets:

• Read the linked conversation before acting on the ticket. The ticket describes the automation's suggested improvement, but the conversation is the source of truth for whether that improvement is right.
• If you believe it's not required, comment why, add the label FIN-REJECTED, and close it as Won't Do.
• If you alter the description significantly for Claude to implement, add the label FIN-MODIFIED.

Oversight

Automation reduces manual review, but doesn't remove it entirely:

• The weekly report (see Automated process above) lists all the low-scoring conversations the run considered, including its reasoning for the ones that didn't get a ticket. Reviewing the report's PR is a lightweight way to check that reasoning each week.
• Each sprint, the docs lead reviews on how Fin is doing overall, which can include looking at high-scoring conversations too, to understand what's going right and not just what's going wrong.

Quick Links

• Fin Analysis Automation GitHub
• Fin Analysis Automation Explainer Doc
• Fin AI dashboard
• Original conversation spreadsheet
• Latest conversation spreadsheet
• Fin Conversation Analyst GPT