On the Multilingual Capabilities of Very Large-Scale English Language Models · HackerPapers discussions

On the Multilingual Capabilities of Very Large-Scale English Language Models — mentioned by famouswaffles

Mon, 29 Apr 2024 19:32:17 GMT

>I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili. A thought experiment from other people comments on another language. So...No. Fabricating failure modes from their personally constructed ideas about how LLMs work seems to be a frustratingly common occurrence in these kinds of discussions. >Frustratingly, just few months ago I read a paper describing how LLMs excessively rely on English-language representations of ideas, but now I…

Paper and discussion history. A mention is not an endorsement.

On the Multilingual Capabilities of Very Large-Scale English Language Models — mentioned by famouswaffles

Fri, 08 Dec 2023 22:51:25 GMT

Language Models trained at the size and data they tested are pretty terrible to begin with, translation or not. Like I said, the crucial thing is that it looks to be a diminishing gap. I'm not saying parallel corpora is doing nothing or is unimportant but that's a clue that monolingual competency of the languages in question is the biggest factor here. The extreme positive transfer language models exhibit in terms of multilingual competency is another clue. https://arxiv.org/abs/2108.13349 Parallel corpora may be a crutch with fading relevance.

Paper and discussion history. A mention is not an endorsement.

On the Multilingual Capabilities of Very Large-Scale English Language Models — mentioned by famouswaffles

Tue, 03 Oct 2023 23:04:03 GMT

>on what basis? What makes a predictor LLM better at translating Latin than a system trained specifically for translation? They just are. Sure it sounds a bit strange if you've never thought about it but they are. >I'm sure they can do a decent job but it's weird to me that someone would leap to GPT-style tech despite its known tendency to hallucinate/make stuff up instead of translation-oriented tools like DeepL or Google Translate 1. They don't just potentially do a decent job. For a couple dozen languages, GPT-4 is by far the best translator you can get your hands on. Google, Deepl are not…

Paper and discussion history. A mention is not an endorsement.

On the Multilingual Capabilities of Very Large-Scale English Language Models — mentioned by famouswaffles

Tue, 11 Jul 2023 15:45:22 GMT

1. LLMs are general pattern machines. They can generalize and complete in context to a wide variety of complex non linguistic patterns.https://general-pattern-machines.github.io/ 2. LLMs see positive transfer in multi lingual capability. For example, an LLM trained on 500B tokens of English and 10B tokens of French will not speak in french like a model trained on only 10B tokens of French. What will happen is that the model will be nearly as competent in French as it is in English https://arxiv.org/abs/2108.13349 3. Language models of code reason better even if the benchmarks have nothing to…

Paper and discussion history. A mention is not an endorsement.

On the Multilingual Capabilities of Very Large-Scale English Language Models — mentioned by famouswaffles

Mon, 10 Apr 2023 18:39:59 GMT

Doesn't really matter. There's lots of positive transfer in individual language learning. Competence in one language bleeds into competence in others. https://arxiv.org/abs/2108.13349 GPT-3 is fluent in many languages despite English taking up 93% of the corpus by word count. French is next with 1.8% https://github.com/openai/gpt-3/blob/master/dataset_statisti... Dunno the statistics of language presence with GPT-4 but it takes it up another level in terms of its multilingual capabilities.

Paper and discussion history. A mention is not an endorsement.

On the Multilingual Capabilities of Very Large-Scale English Language Models — mentioned by famouswaffles

Tue, 14 Mar 2023 19:26:09 GMT

They don't perform poorly (depends on how much corpus training the language received). https://arxiv.org/abs/2108.13349. The important thing is you don't need an equivalent number of tokens for an equivalent performance. So even though GPT-3 only had 1.8 % french by word count, it can speak fluent french. There's positive transfer between languages. Also GLM-130B trained on equal English/Chinese (200b tokens each. 400b total) speaks both just fine.

Paper and discussion history. A mention is not an endorsement.