Tonic Validate is now on GitHub Marketplace! (Part 2)

Ethan P
March 7, 2024
Tonic Validate is now on GitHub Marketplace! (Part 2)
In this article

    Tonic Validate is a free, open-source library for evaluating RAG and LLM based applications. It makes it easy to understand how the answer quality of your application changes over time as parameters are tuned, components are changed, and data sources are refreshed.

    We recently announced a new listing on GitHub Marketplace that provides a GitHub Actions template to run Tonic Validate against code changes on every commit. Today, we’re following up with an additional listing that allows you to establish integration tests each time a branch is merged into your main branch. Together, these two GitHub Actions templates will enable your teams to create scalable CI/CD processes for your RAG systems. It’s available now on GitHub Marketplace, here.

    To get started, follow the instructions found in the marketplace listing. I’ll also give a brief demonstration of how to set up below.

    Setting up Tonic Validate as a GitHub Action

    To use the action in your own GitHub repositories, simply add an additional workflow file in your .github directory. The workflow file will look something like this:

    You must provide just two pieces of information (the project ID and LLM responses) as well as the API keys.

    API Key

    Validate uses an LLM to help evaluate your application. By default, we use GPT-4 Turbo, available from OpenAI. You just need to provide your OpenAI API Key in the workflow file. If you prefer to use Azure OpenAI, then you can instead provide both the equivalent Azure API Key and the Azure OpenAI Endpoint URL.

    Today, Validate supports either OpenAI or Azure OpenAI models, but if there are other LLMs you think we should be using for evaluation, you can file an issue on our main repository.


    Validate takes a set of user-provided questions on which to perform the evaluation. Each question is represented by a JSON object that looks like the below JSON.

    The optional fields shown above allow Validate to compute additional metrics. If those fields are not provided, Validate just computes the metrics that do not require the missing fields.

    The file should ultimately look something like this:

    You check this JSON into your repo. In the YAML, specify the path to the file as the value of the variable llm_response_path.

    Running your action

    Once this setup is complete, your action triggers each time code is merged to main. After the evaluation completes, you can see your results in the Tonic Validate UI as a new data point on each metric visualization.

    A gif showing the Tonic Validate UI, displaying results over time.

    Next Steps

    If you have any feedback, or see a missing feature that you need, just reach out. The easiest way to reach us is through GitHub—file an issue on either the GitHub Action repo or the main Tonic Validate repo. Alternatively, you can check out our UI offering for business, which provides a nice UI experience on top of our open source offering. You can learn more from the Validate page on our website.

    Ethan P
    Software Engineer
    Ethan is a software engineer at Tonic AI. He has a degree in Computer Science with a minor in Math from NYU. He has experience working on machine learning systems for robotics and finance. He also founded a not for profit that builds affordable space satellites.