HellaSwag: Can a Machine Really Finish Your Sentence? · HackerPapers discussions

HellaSwag: Can a Machine Really Finish Your Sentence? — mentioned by recursivecaveat

Sun, 10 Dec 2023 01:32:49 GMT

They used to be quite bad at common-sense-reasoning type tasks like HellaSwag: https://arxiv.org/pdf/1905.07830.pdf but it seems like they're creeping up into the high-80s for % of responses correct now.

Paper and discussion history. A mention is not an endorsement.

HellaSwag: Can a Machine Really Finish Your Sentence? — mentioned by MacsHeadroom

Wed, 29 Mar 2023 03:52:18 GMT

I don't understand what you want me to ask it about. So I asked it something, as usual every citation and URL is real despite not having internet access or training data after 2021: <PROMPT>List some citations of recent Machine Learning papers discussing reasoning ability in Machine Learning Models such as HellaSwag. [REPLY]" As of my knowledge cutoff in September 2021, I can provide you with some citations for ML papers that discuss reasoning abilities in models and datasets like HellaSwag. However, you may want to search for more recent papers using a platform like arXiv or Google Scholar.…

Paper and discussion history. A mention is not an endorsement.

HellaSwag: Can a Machine Really Finish Your Sentence? — mentioned by namuol

Sun, 19 Mar 2023 03:18:56 GMT

FWIW, the “HellaSwag” benchmark from the GPT-4 paper consists of many such “riddles” and it achieves ~95% accuracy (essentially par with humans). The next best model (LLM or otherwise) was around 85%. It is pretty remarkable. GPT-4 paper: https://arxiv.org/pdf/2303.08774.pdf Original HellaSwag paper from 2019: https://arxiv.org/pdf/1905.07830.pdf

Paper and discussion history. A mention is not an endorsement.

HellaSwag: Can a Machine Really Finish Your Sentence? — mentioned by rnosov

Sun, 05 Mar 2023 10:37:33 GMT

You can read the original LLaMA paper which is pretty accessible[1]. For example, they claim to outperform GPT-3 on HellaSwag benchmark ( finishing sentences ). You can find examples of unfinished sentences in the HellaSwag paper [2] on page 13. Unfortunately for LLaMA, most people would be probably just asking questions about Captain Picard and so on, and on this benchmark LLaMA significantly underperforms compared to OpenAI models (thats's from their paper). [1] https://research.facebook.com/file/1574548786327032/LLaMA--O... [2] https://arxiv.org/pdf/1905.07830.pdf

Paper and discussion history. A mention is not an endorsement.