Large Language Monkeys: Scaling Inference Compute with Repeated Sampling · HackerPapers discussions

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — mentioned by quantisan

Thu, 12 Sep 2024 21:39:19 GMT

Amazing! OpenAI figured out how to scale inference. https://arxiv.org/abs/2407.21787 show how using more compute during inference can outperform much larger models in tasks like math problems I wonder how do they decide when to stop these Chain of Thought for each query? As anyone that played with agents can attest, LLMs can talk with themselves forever.

Paper and discussion history. A mention is not an endorsement.

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — mentioned by wesleyyue

Thu, 12 Sep 2024 20:13:45 GMT

Just added o1 to https://double.bot if anyone would like to try it for coding. --- Some thoughts: * The performance is really good. I have a private set of questions I note down whenever gpt-4o/sonnet fails. o1 solved everything so far. * It really is quite slow * It's interesting that the chain of thought is hidden. This is I think the first time where OpenAI can improve their models without it being immediately distilled by open models. It'll be interesting to see how quickly the oss field can catch up technique-wise as there's already been a lot of inference time compute papers recently…

Paper and discussion history. A mention is not an endorsement.

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — mentioned by bluecoconut

Thu, 12 Sep 2024 19:15:27 GMT

Some other follow up reflections 1. I wish that Y-axes would switch to be logit instead of linear, to help see power-law scaling on these 0->1 measures. In this case, 20% -> 80% it doesn't really matter, but for other papers (eg. [2] below) it would help see this powerlaw behavior much better. 2. The power law behavior of inference compute seems to be showing up now in multiple ways. Both in ensembles [1,2], as well as in o1 now. If this is purely on decoding self-reflection tokens, this has a "limit" to its scaling in a way, only as long as the context length. I think this implies (and I am…

Paper and discussion history. A mention is not an endorsement.

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — mentioned by bluecoconut

Thu, 12 Sep 2024 17:54:50 GMT

This power law behavior of test-time improvement seems to be pretty ubiquitous now. In more agents is all you need [1], they start to see this as a function of ensemble size. It also shows up in: Large Language Monkeys: Scaling Inference Compute with Repeated Sampling [2] I sorta wish everyone would plot their y-axis with logit y-axis, rather than 0->100 accuracy (including the openai post), to help show the power-law behavior. This is especially important when talking about incremental gains in the ~90->95, 95->99%. When the values (like the open ai post) are between 20->80, logit and linear…

Paper and discussion history. A mention is not an endorsement.

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — HN submission

Tue, 03 Sep 2024 21:11:48 GMT

Help Finding LLM and Proof Based Refactoring Reference

Recently on HN I came across an LLM paper achieving impressive results on SWE-bench refactoring tasks. In addition to generating the refactoring, it also created a formal proof of equivalence in a special DSL. Now I can't seem to find the paper! So far, I have scrubbed the HN SWE-bench articles from the last 9 months, and tried Google scholar searches for LLM based refactoring with Proofs or DSLs. I am confident that they mention SWE-bench in the abstract, but it does not seem to be mentioned in the HN announcement. I don't remember if it was a custom DSL, or if they used a well known…

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — mentioned by charles_irl

Tue, 06 Aug 2024 17:46:25 GMT

One of the authors, Charles, here. I wrote this up because 1) I've been following work on sampling/inference-time compute scaling and the "Large Language Monkeys" paper (https://arxiv.org/abs/2407.21787) results looked promising and 2) our reproduction was effortless and immediate, which has not been my experience with most research. Happy to answer questions!

Paper and discussion history. A mention is not an endorsement.

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — HN submission

Sun, 04 Aug 2024 13:12:54 GMT

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — HN submission

Thu, 01 Aug 2024 17:49:40 GMT

Beating GPT-4o and Claude 3.5 on SWE-bench Lite through repeated sampling

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — HN submission

Thu, 01 Aug 2024 16:28:59 GMT

Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.