A Benchmark for Learning to Translate a New Language from One Grammar Book · HackerPapers discussions

A Benchmark for Learning to Translate a New Language from One Grammar Book — mentioned by glomgril

Tue, 15 Apr 2025 00:32:34 GMT

Check out this recent benchmark MTOB (Machine Translation from One Book) -- relevant to your comment, though the book does have parallel passages so not exactly what you have in mind: https://arxiv.org/pdf/2309.16575 In the case of non-human communication, I know there has been some fairly well-motivated theorizing about the semantics of individual whale vocalizations. You could imagine a first pass at something like this if the meaning of (say) a couple dozen vocalizations could be characterized with a reasonable degree of confidence. Super interesting domain that's ripe for some fresh…

Paper and discussion history. A mention is not an endorsement.

A Benchmark for Learning to Translate a New Language from One Grammar Book — mentioned by visarga

Wed, 06 Nov 2024 18:10:18 GMT

I don't think "humans/animals learn faster" holds. LLMs learn new things on the spot, you just explain it in the prompt and give an example or two. A recent paper tested both linguists and LLMs at learning a language with less than 200 speakers and therefore virtually no presence on the web. All from a few pages of explanations. The LLMs come close to humans. https://arxiv.org/abs/2309.16575 Another example is the ARC-AGI benchmark, where the model has to learn from a few examples to derive the rule. AI models are closing the gap to human level, they are around 55% while humans are at 80%.…

Paper and discussion history. A mention is not an endorsement.

A Benchmark for Learning to Translate a New Language from One Grammar Book — HN submission

Wed, 26 Jun 2024 04:07:24 GMT

A Benchmark for Learning to Translate a New Language from One Grammar Book

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

A Benchmark for Learning to Translate a New Language from One Grammar Book — mentioned by abetusk

Wed, 26 Jun 2024 03:04:43 GMT

The paper in question is "A Benchmark for Learning to Translate a New Language from One Grammar Book" by Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, Luke Melas-Kyriazi [0]. [0] https://arxiv.org/abs/2309.16575

Paper and discussion history. A mention is not an endorsement.

A Benchmark for Learning to Translate a New Language from One Grammar Book — mentioned by sanxiyn

Sun, 31 Mar 2024 18:24:09 GMT

Considering MTOB(Machine Translation from One Book) result, where LLM in-context learns a new language from a grammar book, I wonder how LLMs fare given, say, GnuCOBOL Programmer's Guide PDF, which is easily available. It would be an interesting addition to the benchmark. https://arxiv.org/abs/2309.16575 https://gnucobol.sourceforge.io/guides.html

Paper and discussion history. A mention is not an endorsement.