Hurdles to Progress in Long-form Question Answering — mentioned by rushingcreek

Tue, 07 Feb 2023 01:16:31 GMT

Ah yes, it's this paper: https://arxiv.org/abs/2103.06332. With recent advances in alignment, this problem has gotten much better. Furthermore, the LLM used in that paper is only ~400 million parameters, which is orders of magnitude smaller than models being used for this task today. Scale + alignment has been the solution so far.

Paper and discussion history. A mention is not an endorsement.

Hurdles to Progress in Long-form Question Answering — mentioned by rushingcreek

Wed, 06 Jul 2022 20:31:10 GMT

I'd be happy to talk a bit about how we evaluated the model. The task we're performing is fundamentally long-form question answering (LFQA), and recent papers (https://arxiv.org/pdf/2103.06332.pdf) have shown that metrics such as ROUGE (used for the KITE benchmark) aren't great at evaluating the quality & truthfulness of generated answers. On our dataset, our approach is to use a combination of human evaluation (which is still arguably the most reliable metric used by the NLP research community to evaluate generated answer quality) and an entailment score (checking if a generated answer is…

Paper and discussion history. A mention is not an endorsement.

Hurdles to Progress in Long-form Question Answering · HackerPapers discussions

Hurdles to Progress in Long-form Question Answering — mentioned by rushingcreek

Hurdles to Progress in Long-form Question Answering — mentioned by rushingcreek