BBQ: A Hand-Built Bias Benchmark for Question Answering · HackerPapers discussions

BBQ: A Hand-Built Bias Benchmark for Question Answering — mentioned by westurner

Tue, 04 Feb 2025 00:24:42 GMT

That would disincentive this type of research, for example: "DeepSeek's Hidden Bias: How We Cut It by 76% Without Performance Loss" (2025) https://news.ycombinator.com/item?id=42868271 https://news.ycombinator.com/item?id=42891042 TIL about BBQ: Bias Benchmark for QA "BBQ: A Hand-Built Bias Benchmark for Question Answering" (2021) https://arxiv.org/abs/2110.08193

Paper and discussion history. A mention is not an endorsement.

BBQ: A Hand-Built Bias Benchmark for Question Answering — mentioned by Manuel_D

Wed, 29 Jan 2025 20:51:18 GMT

It's from the bias set linked in the article: https://arxiv.org/abs/2110.08193

Paper and discussion history. A mention is not an endorsement.

BBQ: A Hand-Built Bias Benchmark for Question Answering — mentioned by JumpCrisscross

Wed, 29 Jan 2025 18:38:07 GMT

They measure bias using "the Bias Benchmark for QA (BBQ), a dataset of question sets...that highlight attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts. Our task evaluates model responses at two levels: (i) given an under-informative context, we test how strongly responses reflect social biases, and (ii) given an adequately informative context, we test whether the model's biases override a correct answer choice" [1]. Aside: I'm curious how distillation affects such scores. If I distill an unbiased…

Paper and discussion history. A mention is not an endorsement.

BBQ: A Hand-Built Bias Benchmark for Question Answering — mentioned by blueboo

Sun, 05 Nov 2023 08:17:50 GMT

There are all sorts. You do have to be specific about what you mean by “bias” though. https://arxiv.org/abs/2110.08193 and https://arxiv.org/abs/1804.09301 and https://arxiv.org/abs/2206.04615 …

Paper and discussion history. A mention is not an endorsement.