On the Measure of Intelligence · HackerPapers discussions

On the Measure of Intelligence — mentioned by surprisetalk

Wed, 16 Sep 2026 13:52:07 GMT

I rather like this paper, but I think it is generous to say their benchmarks measure "intelligence". [0] https://arxiv.org/pdf/1911.01547 We have no dang clue what intelligence is, nor how to measure it. [1] https://taylor.town/crowpower

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — HN submission

Fri, 04 Sep 2026 13:57:59 GMT

The Measure of Intelligence (2019)

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — mentioned by astrobiased

Thu, 03 Sep 2026 21:11:23 GMT

I can’t help but notice how much this echoes Francois Chollet’s On the Measure of Intelligence: https://arxiv.org/abs/1911.01547 Most of frontier-model progress still looks like skill acquisition optimization: broader benchmark coverage and performance, more domains absorbed into the training distribution, and increasingly strong performance within that surface area. It seems more about coverage-driven competence. Somewhat analogous to overfitting at scale. The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new? With our current AI…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — HN submission

Thu, 13 Aug 2026 06:20:50 GMT

The Measure of Intelligence

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — mentioned by LarsDu88

Wed, 12 Aug 2026 20:00:02 GMT

You're take basically lines up with Francois Chollet: https://arxiv.org/abs/1911.01547 intelligence is more like polishing a ball smooth than growing the ball to infinity. For many tasks, it will be smooth enough.

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by gobdovan

Sat, 30 May 2026 17:53:15 GMT

Yeah, but there's attempts to fix it. The Cholet paper (https://arxiv.org/abs/1911.01547) is a good attempt, shifting from measuring ~skill to measuring ~acquiring skill. It's the framework behind ARC-AGI benchmarks.

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by daveguy

Fri, 27 Mar 2026 19:58:34 GMT

Just because humans are usually tested in a particular way that allows them to make up for a lack of generality with an outstanding performance in their specialization doesn't mean that is a good way to test generalization itself. Apparently someone here doesn't know how outliers affect a mean. Or, for that matter, have any clue about the purpose of the ARC-AGI benchmark. For anyone who is interested in critical thinking, this paper describes the original motivation behind the ARC benchmarks: https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by daveguy

Thu, 26 Mar 2026 21:36:28 GMT

It may have been tested on the full set, but the score you quote is for a single game environment. Not the full public set. That fact is verbatim in what you responded to and vbarrielle quoted. It scored 97% in one game, and 0% in another game. The full prelude to what vbarrielle quoted, the last sentence of which you left out, was: > We then tested the harnesses on the full public set (which researchers did not have access to at the time). We found extreme bimodal performance across the two sets, controlling for the same frontier model... The harness only transfers to like-environments and…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by cubefox

Mon, 07 Jul 2025 23:30:36 GMT

> We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans. https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by cubefox

Mon, 07 Jul 2025 21:14:21 GMT

> Getting a high score on ARC doesn't mean we have AGI and Chollet has always said as much AFAIK He only seems to say this recently, since OpenAI cracked the ARC-AGI benchmark. But in the original 2019 abstract he said this: > We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans. https://arxiv.org/abs/1911.01547 Now he seems to backtrack, with the release of harder ARC-like benchmarks, implying that the first one didn't actually test for really general human-like…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by nxobject

Mon, 07 Jul 2025 13:07:26 GMT

I understand Chollet is transparent that the "branding" of the ARC-AGI-n suites is meant to be suggestive of its purpose, than substantial. However, it does rub me the wrong way - as someone who's cynical of how branding can enable breathless AI hype by bad journalism. A hypothetical comparison would be labelling SHRDLU's (1968) performance on Block World planning tasks as "ARC-AGI-(-1)".[0] A less loaded name like (bad strawman option) "ARC-VeryToughSymbolicReasoning" should capture how the ARC-AGI-n suite is genuinely and intrinsically very hard for current AIs, and what progress…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by cubefox

Mon, 07 Jul 2025 13:05:53 GMT

ARC-AGI was introduced in 2019: https://arxiv.org/abs/1911.01547 GPT-3 didn't come out until 2020.

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by whiplash451

Mon, 07 Jul 2025 12:34:41 GMT

You're not the only one. ARC-AGI is a laudable effort, but its fundamental premise is indeed debatable: "We argue that human cognition follows strictly the same pattern as human physical capabilities: both emerged as evolutionary solutions to specific problems in specific evironments" (from page 22 of On the Measure of Intelligence) https://arxiv.org/pdf/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by mrandish

Thu, 23 Jan 2025 21:10:10 GMT

Assessing AI's progress toward replicating the full breadth and depth of human intelligence is a deceptively hard problem. A paper by François Chollet, who was until recently a researcher at Google, called "On the Measure of Intelligence" is the best overview of the challenges I've read. Highly recommended. https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — HN submission

Sat, 21 Dec 2024 03:58:17 GMT

On the Measure of Intelligence

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — mentioned by not-a-cat

Thu, 03 Oct 2024 18:47:39 GMT

Unnamed AI research lab | ML research engineer | Zurich, Switzerland | Full-time | Onsite | 100-200k CHF I am starting a research lab funded by a hedge fund that I created 5 years ago. The hedge is doing well, and applies ML to quant finance. The research and trading activities should reinforce each other over time: the research publishes research and fosters in-house ML expertise. The trading applies ML to quant finance and brings funding. The research entity is incorporated, funded, has an office and 0 employees currently. My vision for the research lab is the following: - A bunch of phds…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by not-a-cat

Sun, 15 Sep 2024 21:18:13 GMT

Unnamed AI research lab | ML research engineer | Zurich, Switzerland | Full-time | Onsite | 100-200k CHF I am starting a research lab funded by a hedge fund that I created 5 years ago. The hedge is doing well, and applies ML to quant finance. The research and trading activities should reinforce each other over time: the research publishes research and fosters in-house ML expertise. The trading applies it to quant finance and brings funding. The research entity is incorporated, funded, has an office and 0 employees currently. My vision for the research lab is the following: - A bunch of phds…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by YeGoblynQueenne

Sun, 15 Sep 2024 11:05:54 GMT

What they mean by "specific learned skill" is that each task illustrates the use of certain "core knowledge priors" that François Chollet has claimed are necessary to solve said tasks. You can find this claim in Chollet's white paper that introduced ARC, linked below: On the Measure of Intelligence https://arxiv.org/abs/1911.01547 "Core knowledge priors" are a concept from psychology and cognitive science as far as I can tell. To be clear, other than Chollet's claim that the "core knowledge priors" are necessary to solve ARC tasks, as far as I can tell, there is no other reason to assume so…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — HN submission

Sat, 15 Jun 2024 06:17:00 GMT

The Measure of Intelligence

16 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — mentioned by mischa_u

Wed, 12 Jun 2024 07:45:34 GMT

"On the Measure of Intelligence" https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by visarga

Mon, 22 Apr 2024 18:55:12 GMT

> The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, concerning priors, experience, and generalization difficulty. (Chollet, 2019, https://arxiv.org/pdf/1911.01547.pdf) Priors here means how targeted is the model design to the task. Experience means how large is the necessary training set. Generalization difficulty is how hard is the task. So intelligence is defined as ability to learn a large number of tasks with as little experience and model selection as possible. If it's a skill only possible because your model already follows the…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — HN submission

Wed, 03 Apr 2024 14:45:53 GMT

On the measure of intelligence (2019)

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — mentioned by levocardia

Mon, 11 Mar 2024 00:18:05 GMT

A little surprised that neither the paper nor the discussion here cite Francois Chollet's 2019 paper on defining intelligence for machines [1], which talks explicitly about visual reasoning and proposed an RPM-like visual task as a benchmark. I found it very useful for thinking about what it means for an AI to be generally intelligent. From what I remember (having not read it in a while), his idea is basically that general intelligence is the efficiency with which you can learn, after appropriately "penalizing" any baked-in priors or knowledge beyond what a typical animal is born with. As an…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by TacticalCoder

Wed, 06 Mar 2024 11:17:26 GMT

> IQ tests for models seem...somewhat flawed. They are for all the reasons you explain. Here's a wonderful paper (which I probably found here on HN) on the subject: On the Measure of Intelligence, by Francois Chollet (working for Google), from 2019: https://arxiv.org/abs/1911.01547 He comes up with many tests which are relatively simple for humans that AI models cannot solve. If you don't want to read all the paper, jump to the figures (starting from page 48). The goal, of course, is not to train an AI on the correct results of the exact problems in that paper: that'd just be more memory…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by TacticalCoder

Wed, 17 Jan 2024 03:20:26 GMT

A paper linked in TFA: https://arxiv.org/pdf/1911.01547.pdf Start at page 48 from this paper to see lots of cool tests, totally trivial for us humans to solve, that apparently AI have issues with. The paper is from late 2019: I wonder how multi-modal current AIs do on these tests.

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by jewelry

Tue, 19 Sep 2023 05:55:50 GMT

Exactly. And this study did a good job exploring it philosophically. https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by sanxiyn

Tue, 04 Apr 2023 05:46:20 GMT

You know, François Chollet literally did this, and people don't listen. People should listen to Chollet more. https://twitter.com/fchollet/status/1638643323748618240 https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — HN submission

Mon, 23 Jan 2023 21:11:22 GMT

On the Measure of Intelligence (2019)

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — mentioned by sinenomine

Thu, 05 Jan 2023 16:48:02 GMT

To add to it: a recent attempt at designing a psychometric test encoding human cognitive priors: https://arxiv.org/abs/1911.01547 - the ARC dataset. https://github.com/google/BIG-bench - the most recent kitchen sink of a cognitive test, which includes the previous test as a subtask. Apparently there are large language models (Chinchilla) that perform comparably to a median human level on this second test.

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by nl

Sun, 16 Oct 2022 05:53:52 GMT

> I don't think we are near general AI Nor do I. But I don't agree with your definition of general AI at all. I think by your definition we are well on the way towards it. I'd note that you didn't address the idea that compression of input concepts into a lower dimensional space is exactly the "reliably turning symbols into meaning" idea you suggest. The fact the latent representations of concepts can be manipulated in ways that make logical sense is a good indication that the symbols have meaning. The classic Word2Vec experiments showing how the relationships Paris->France ~= London->England…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by visarga

Thu, 15 Sep 2022 15:01:35 GMT

> makes the best possible decision about the future with the information that it has available So would a good thermostat be considered intelligent, assuming it takes great decisions? Intelligence needs to be defined with respect to a range of tasks and a range of priors. It's skill acquisition efficiency, not just taking good decisions. I recommend "On the Measure of Intelligence" by François Chollet >We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by visarga

Tue, 05 Jul 2022 18:57:53 GMT

> Intelligence is the ability to achieve goals A more precise definition by François Chollet: The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty https://arxiv.org/pdf/1911.01547.pdf So a system is more intelligent if it can solve harder tasks with fewer trials and prior knowledge. Intelligence is always defined over a scope of tasks, for example human intelligence only applies to the space of tasks and domains that fit within the human experience. Our intelligence does not have…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by visarga

Wed, 01 Dec 2021 21:15:34 GMT

Ability to solve novel problems with little experience. Skill acquisition efficiency. On the Measure of Intelligence, François Chollet https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by yamrzou

Sun, 06 Jun 2021 13:05:04 GMT

Very Interesting. This reminds me of The Abstraction and Reasoning Corpus [1] by François Chollet, accompanying his paper On the Measure of Intelligence [2]... Edit: Found a recent article mentioning both and discussing a NeurIPS paper on using Bongard problems to test AI systems [3]. [1] https://github.com/fchollet/ARC [2] https://arxiv.org/abs/1911.01547 [3] https://spectrum.ieee.org/tech-talk/artificial-intelligence/...

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by nl

Fri, 30 Apr 2021 03:30:43 GMT

In the HANS paper I'd point out this bit: "we retrained each model on the MNLI training set augmented with a dataset structured exactly like HANS (i.e. using the same thirty subcases) but containing no specific examples that appeared in HANS.... In general, the models trained on the augmented MNLI performed very well on HANS" I think it's great to find examples where language models break. But it doesn't look like they have found a fundamental problem here - just a weakness that can be fixed with "just" engineering. I do agree there are things we don't know how to do yet. I think Chollet's…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — HN submission

Sat, 03 Apr 2021 07:26:31 GMT

On the Measure of Intelligence, Chollet [pdf]

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — mentioned by m3at

Wed, 09 Dec 2020 01:11:25 GMT

On the Measure of Intelligence, François Chollet [1] Fellow HNer seems to have liked a lot of ML paper, this is not breaking the trend. This is a great meta paper questioning the goal of the field itself, and proposing ways to formally evaluate intelligence in a computational sense. Chollet is even ambitious enough to propose a proof of concept benchmark! [2] I also like some out of the box methods people tried to get closer to a solution, like this one combining cellular automata and ML [3] [1] https://arxiv.org/abs/1911.01547 [2] https://github.com/fchollet/ARC [3]…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by nl

Wed, 25 Nov 2020 10:32:18 GMT

I think GPT-3 is amazing, and the equal of humans in many cases. But I think that this is because in many cases humans speak based on reflex and without thought. GPT-3 is incapable of abstract thought. In terms of actual measurable tests it doesn't pass the kinds of tests proposed in Francois Chollet's "On the Measure of Intelligence": https://arxiv.org/pdf/1911.01547.pdf

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by YeGoblynQueenne

Sat, 21 Nov 2020 01:49:57 GMT

Claims 78.8% on the Abstract Reasoning Corpus [1] (ARC) in the abstract, but clarifies that this is only on 100 out of the 400 evaluation tasks, all on 10 x 10 grids. The 10x10 grids are hard-coded in the network architecture so it's not possible for the system to perform any better without re-design and re-training. Trains on a dataset augmented to 1500 tasks by permuting colours and "exploiting that the tasks are invariant to rotation and symmetry". So it's cheating, but that just goes to show that the ARC dataset is not the full-proof benchmark of intelligence that its author meant it to…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by YeGoblynQueenne

Fri, 20 Nov 2020 11:00:26 GMT

Deep neural nets do not learn only from examples! They encode strong inductive biases in their carefully hand-engineered and hand-tuned architectures, hence for example CNNs are used for image recognition and LSTMs for sequence learning etc. Without these biases deep neural nets would not be able to generalise as well as they do (in the sense of local generalisation but not global generalisation as meant by François Chollet [1]). The biggest advances in deep neural nets have come from the discovery and use of good inductive biases: training with gradient descent, backpropagation, more hidden…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by visarga

Sun, 19 Jul 2020 04:50:14 GMT

The measure of intelligence, François Chollet https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by visarga

Tue, 09 Jun 2020 18:32:20 GMT

> There is a lot of debate over the definition of intelligence but one thing most definitions have in common is that it is general. A human being can play the piano, solve an integral, write a novel, compose a photograph, and many many more things. There is a nice paper on this topic. On the Measure of Intelligence - François Chollet https://arxiv.org/abs/1911.01547 I believe generality in human intelligence is overrated. We can't do large multiplication in our heads, we can't solve the Salesman problem in our heads, and many other things we can't do because our intelligence is not in fact…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by andra1

Mon, 08 Jun 2020 21:47:12 GMT

Francois Chollet tries to address some of these problems by tackling how intelligence is measured. Most AI systems are measured by task performance, but he argues that intelligence is about "skill-acquisition efficiency". https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by hardmaru

Fri, 14 Feb 2020 05:42:47 GMT

Based on "On the Measure of Intelligence" https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by Isinlor

Sat, 23 Nov 2019 15:24:13 GMT

If someone wants to go as far as to claim that any computation is just curve-fitting then your statement is equivalent to Church–Turing thesis. There are no formal arguments against Church–Turing thesis. From that perspective intelligence is indeed just a curve fitting. https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis I really enjoyed the "The Measure of Intelligence" by François Chollet. https://arxiv.org/abs/1911.01547 "We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at…

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — mentioned by glial

Mon, 18 Nov 2019 16:07:03 GMT

I'm not sure this is necessarily true. At a smaller scale, we didn't need to understand how humans play chess in order to build a good chess AI. I do think we need a better understanding of what we mean when we talk about intelligence, if we expect to recreate it. Francois Chollet has a good recent paper about this [1]. [1] https://arxiv.org/abs/1911.01547

Paper and discussion history. A mention is not an endorsement.

On the Measure of Intelligence — HN submission

Fri, 15 Nov 2019 19:31:20 GMT

The Measure of Intelligence: towards more human-like artificial systems

80 points; 12 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — HN submission

Thu, 14 Nov 2019 07:41:20 GMT

The Measure of Intelligence

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — HN submission

Wed, 13 Nov 2019 01:48:36 GMT

The Measure of Intelligence by François Chollet

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Measure of Intelligence — HN submission

Sat, 09 Nov 2019 11:35:23 GMT

The Measure of Intelligence

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.