Test your prompts, models, and RAGs. Catch regressions and improve prompt quality. LLM evals for OpenAI, Azure, Anthropic, Gemini, Mistral, Llama, Bedrock, Ollama, and other local & private models with CI/CD integration.

A framework for few-shot evaluation of language models.

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

Python SDK for running evaluations on LLM generated responses

Python client for Kolena's machine learning testing platform

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.

The LLM Evaluation Framework

Expressive is a cross-platform expression parsing and evaluation framework. The cross-platform nature is achieved through compiling for .NET Standard so it will run on practically any platform.

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

Evaluating LLMs' Cognitive Behavioral Reasoning for Cybersecurity

Open-Source Evaluation for GenAI Application Pipelines

Test and evaluate LLMs and model configurations, across all the scenarios that matter for your application

The Core Reinforcement Learning library is intended to enable scalable deep reinforcement learning experimentation in a manner extensible to new simulations and new ways for the learning agents to interact with them. The hope is that this makes RL research easier by removing lock-in to particular simulations.The work is released under the follow…

Evaluation framework for oncology foundation models (FMs)

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

Official Implementation of ACL2024 paper "Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs"(https://arxiv.org/abs/2402.11199).

Evaluate the quality of SRT files using the multilingual multimodal SONAR model.

Evaluate custom and HuggingFace text-to-image/zero-shot-image-classification models like CLIP, SigLIP, DFN5B, and EVA-CLIP. Metrics include Zero-shot accuracy, Linear Probe, Image retrieval, and KNN accuracy.

Evaluation of label free cell and nuclei segmentation

PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection