Datakami helped a SaaS startup turn an LLM prototype into an evaluated, production-ready AI feature. We collaborated with end-users to build trust in the final product from the start.
Our client is a SaaS startup with a high-touch onboarding flow. Their customers supply reference documents, which an in-house team of specialists turns into material for the client's platform. Demand was outgrowing what the internal team could handle, which was a bottleneck to further growth. The company wanted to automate the onboarding as much as possible and let customers generate their own platform materials instead. The company also expected that that a high-quality AI onboarding flow could help with securing more funding.
Their sole ML engineer had built a collection of promising LLM prototypes in Jupyter notebooks. But the ML engineer was blocked on a bunch of architectural decisions, and had no other team members to discuss this with. Which model should we use? How do we know the outputs are good? How do we get the rest of the company to trust this automation, especially the domain experts that we're building the system for?
Turning heterogeneous customer documents (PDFs, slides, compliance documents, policy documents) into high-quality platform materials is not a task that can be solved with a single prompt. The client needed:
Observability first. As always, we started by making all generative AI behaviour visible. We collected the client's requirements for an LLM observability tool (self-hosted, long trace retention), selected Langfuse v2 as the best fit, deployed it, and connected the existing prototypes. Langfuse also became the team's prompt management system and experiment platform. We started building an evaluation set and put it in Langfuse, so the same dataset could be re-run across different models, parameters and prompts.
A stronger pipeline. We used the observability data to redesign the LLM workflow from a single-shot approach to a two-step pipeline. This had as benefit that each step could be evaluated and optimized separately, which significantly improved output quality on heterogeneous documents. We tested multiple LLMs on a realistic dataset for the client's task, introduced vision-LLMs for extracting more information from complex customer PDFs, and ported the ML engineer's fragmented notebook experiments into a single production-ready codebase.
Evaluation with the experts in the loop. Datakami founder Judith van Stegeren drew on her research background to design a formal evaluation approach. In a series of facilitated evaluation sessions, she worked with the domain experts to explain what evaluation is, how it works, and how they should annotate LLM outputs. The team came away enthusiastic and committed to grading a statistically significant sample of outputs using a purpose-built grading sheet.
Automated quality gates. Together with the client's ML engineer, we defined a list of output requirements for the generated platform content. Valid JSON for downstream processing, appropriate reading level, length limits, refusal flagging, and more. We implemented automated tests and trace scoring to detect changes in output quality and regressions, and facilitate future experiments.
Full knowledge transfer. All technical and design decisions were documented in the client's own Notion workspace. When we left, everything we investigated, designed and built was already in their systems. At the end of the project, the client's employees owned everything: production-quality code, their own Langfuse instance, an evaluation routine including a dataset and grading sheets. During the retrospective, the client's engineer commented on the value of this project for his professional development.
Within a few weeks, the client went from notebook experiments to a working, monitored production-ready pipeline. Before the project, there was a promising prototype and many open questions. After it, they had tracing and prompt management in Langfuse, automated quality checks on every output, and a repeatable evaluation process that the team can run themselves. The client's ML engineer is unblocked.
"I knew that with Datakami I would get great results. They are experts, they know what they're doing, and they actually implement it."
As a bonus, the client's internal team of specialists understands how the system works and directly influenced its design.
The internal ML engineer also learned some new things along the way.
"I have seen so many best practices! How the prompts are written, the evaluation and refactoring cycle, how to get access to all these different models. Development-wise, it was very good value for me personally."
After the project, the front-end team took over to integrate the pipeline into the main platform. The ML engineer continued working on the production pipeline. In the final weeks of the project we brainstormed potential improvements and extensions together.
Once the new AI system has been integrated, the domain experts can spend their time white-glove onboarding the trickiest customers of the client company.
If your team has a promising LLM prototype but production feels far away, or if you're doing evals for the first time and want to do them right: that's our sweet spot. Book a call and tell us what you need.