Adding AI to document and email workflows safely

Where AI fits in a document workflow

Large language models are good at reading messy, inconsistent text and producing something structured. That maps onto four jobs that show up in almost every back office:

What these have in common is that the output can be checked, by code, by a person, or both. Tasks where the model makes a decision nobody reviews, or takes an action outside your systems, belong later, once you've seen how it performs on your documents.

A reference design

The model is one step in a pipeline, not the pipeline. Most of the engineering, and most of the safety, is in the steps around it.

• Intake. Read from a shared mailbox through Microsoft Graph or your mail platform's API, or from a monitored folder or bucket. Use an app registration or service account scoped to that mailbox or folder only; Exchange Online, for example, can limit an app's mail permissions to named mailboxes.
• Extraction. Get clean text before the model sees it. Many PDFs contain real text; scanned ones need OCR. Recent models can also read images and PDFs directly, which is worth testing on your worst documents.
• Structured output. Ask for a specific JSON shape. The major providers each offer a way to request output that matches a schema, which removes a whole class of parsing errors.
• Validation. Check the output with ordinary code: required fields present, part numbers that exist in your item master, dates that make sense, totals that add up.
• Routing. Results that pass go straight through; anything uncertain or failing goes to a person.

Human review that actually gets used

"Human in the loop" only works if the review step is faster than doing the work by hand. Design the review screen with the same care as the automation.

• Show the source document next to the extracted fields, with the relevant passage highlighted where possible.
• Let reviewers correct a field in place rather than re-keying the record.
• Save every correction. Corrections are the best test cases you'll ever get.
• Start in suggestion mode, where people approve everything, and let categories graduate to straight-through processing only once their measured accuracy justifies it.

Data handling and security

Your security team will ask where the data goes, who can see it, and what the provider does with it. Have the answers written down before the first real document is sent.

Choosing between Claude, OpenAI, Gemini and Grok

We've integrated all four of these providers' APIs. Our honest view is that the right model depends on your documents, and that the differences between providers change often enough that any ranking written today will be stale soon. What stays true:

• Test on your documents. Run the same evaluation set through two or three candidates. Scanned forms, long email threads and spreadsheets-as-PDFs expose differences quickly.
• Weigh hosting and terms alongside quality. Availability through a cloud you already use, regional options and contractual terms can matter more than a small accuracy difference.
• Use the smallest model that passes. Many classification and extraction tasks run well on faster, cheaper model tiers; save the largest models for hard summarization or reasoning.
• Keep a thin abstraction. Put the provider call behind one internal interface so you can switch models or run two side by side without rewriting the workflow.

Our own product, Round Table, puts Claude, ChatGPT, Grok and Gemini in one conversation, with streaming responses and usage tracking (token accounting) built in. Building it is where much of our view on provider differences and usage comes from.

Evaluate before launch and after every change

A demo on five hand-picked emails tells you nothing about the five hundred you'll see next month. Evaluation is what turns "it seems to work" into a number you can manage.

1. 1Collect a representative sample of past documents, including the ugly ones: forwards, replies, scans and multi-language messages.
2. 2Record the correct answer for each, ideally from the people who do the work today.
3. 3Measure accuracy per category and per field, not just overall. A single field that's often wrong can sink a workflow that looks fine on average.
4. 4Rerun the set whenever you change the prompt, the model or the extraction step, and compare against the last run before releasing.
5. 5Add every correction from the review queue to the set, so it grows with the real mix of documents.
6. 6Track cost per document alongside accuracy, so a model change that's slightly better but much more expensive is a visible decision.

A sensible rollout

Pick one document type with clear volume and a clear owner. Run the pipeline in shadow mode alongside the current process and compare. Move to suggestion mode with review of everything, then let the categories that measure well go straight through. Only then consider the next document type, reusing the same intake, validation, review and logging pieces.

Our experience includes document and email processing, classification, summarization and automated workflows with AI. If the steps change from case to case, putting AI agents to work on internal processes covers when an agent fits better than a pipeline. For how we'd approach yours, see AI integration; if your developers want to use AI tools in their own work, AI Velocity Training is the related offer.

Sources

• Role Based Access Control for Applications in Exchange Online (Microsoft)
• PDF and file input: Claude, OpenAI, Gemini
• Structured output: Claude, OpenAI, Gemini, xAI
• Cloud hosting: Claude on Amazon Bedrock, Claude on Vertex AI, models on Vertex AI, Azure OpenAI
• LLM01: Prompt injection (OWASP GenAI Security Project)