SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines · HackerPapers discussions

SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines — mentioned by sourabh03agr

Tue, 23 Jan 2024 16:27:35 GMT

Operationalizing large language models (LLMs) is challenging, mainly due to their unpredictable behaviors and potentially catastrophic failures. A few of the essential requirements for productionizing LLM applications are evaluating them for a variety of checks as well as monitoring their performance in production. This process carries two major friction points: 1. Identifying which evaluations to run is tricky: LLM Prompts are growing in complexity. Prompts nowadays have 50+ instructions with many repetitions. SPADE, developed by researchers at UC Berkeley, HKUST, LangChain, and Columbia…

Paper and discussion history. A mention is not an endorsement.

SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines — mentioned by typpo

Thu, 18 Jan 2024 17:14:52 GMT

Congrats on the launch! I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo (https://github.com/typpo/promptfoo), but it is non-RAG so more simplistic than your implementation. Was also eyeballing this paper https://arxiv.org/abs/2401.03038, which outlines a method for generating asserts from prompt version history that may also be useful for these eval tools.

Paper and discussion history. A mention is not an endorsement.