Building a test strategy to your LLM app

If you build (and don’t test), they will come (and churn)!

There are many reasons why you should be serious about building a test strategy for your LLM app:

• If you don’t find bugs, your customers or users will.
• Post-release debugging is the most expensive form of development.
• Buggy software hurts operations, sales, and reputation.

There are various testing approaches available.

Here are 3 principles for building a solid testing strategy for your LLM app:

1) Make it risk-focused.

A good test strategy is risk-focused.

It's hard and expensive to test everything.

So extend your test coverage to address what matter the most.

Testing is prioritized in terms of the probability that some features of the application will fail and the probability of failure. The greater the probability of an expensive failure, the more important it is to test that use case as early and as carefully as possible.

Look at a product and ask: how can it fail? What can go terribly wrong here and cost my reputation, my cash?

Start building test cases to address these aspects.

2) Diversify approaches in your test strategy

There is a variety of strategies and approaches.

There are many techniques for you to test your LLM-based app. You don’t need to pick only. For some use cases, prompting a LLM will do. To others, it might be very expensive, a regex will do.

Be open to consider different evaluators.

3) Be pragmatic

Don’t suggest test strategies that are far beyond the capabilities of the project.

Start small and grow incrementally.

Don’t include activities on your plate unless they address a risk that matters enough to spend time testing.

Conclusion

Testing is crucial for your LLM app. It prevents customer loss, expensive debugging, and reputational damage. Focus on critical risks, mix up your testing methods, and keep it lean.