GPT-4 Technical Report

1. Why do LLM pelicans look like this? Oct 20261 comment

   • > Interesting. I'd love to see a paper on this. The first time I personally heard of RLHF ruining some model qualities was actually in the GPT-4 paper (march 2023); figure 8 there shows that the base model has near-perfect calibration on multiple-choice questions (!) and this…

     stratos123Oct 15, 2026View on HNParent context

2. Microgpt Mar 20261 comment

   • The model could report the confidence of its output distribution, but it isn't necessarily calibrated (that is, even if it tells you that it's 70% confident, it doesn't mean that it is right 70% of the time). Famously, pre-trained base models are calibrated, but they stop being…

     andy12_Mar 1, 2026View on HNParent context

3. Show HN: Semantic Calculator (king-man+woman=?) May 20251 comment

   • > The results are surprisingly good, I don't think I could've done better as a human I'm actually surprised that the performance is so poor and would expect a human to do much better. The GPT model has embedding PLUS a whole transformer model that can untangle the embedded…

     godelskiMay 15, 2025View on HNParent context

4. Recent results show that LLMs struggle with compositional tasks Feb 20251 comment

   • In addition to passing bar exam[0], improved performance on medical questions[1], economics questions that experts thought it was years away from[2], all the other things marked in green on page 6 were just the changes from 3.5 to 4: https://arxiv.org/pdf/2303.08774 4o added…

     ben_wFeb 2, 2025View on HNParent context

5. Entropy of a Large Language Model output Jan 20251 comment

   • You are observing "flattened logits" https://arxiv.org/pdf/2303.08774#page=12&org=openai . The entropy of ChatGPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens"…

     gwernJan 13, 2025View on HNParent context

6. Reasoning in Large Language Models: A Geometric Perspective Jul 20241 comment

   • I think you're close enough that the differences probably aren't too important. But if you want a bit more nuance, then read on. For disclosure, I'm in the second camp here. But I'll also say that I have a lot of very strong evidence to support this position, and that I do this…

     godelskiJul 7, 2024View on HNParent context

7. Creativity has left the chat: The price of debiasing language models Jun 20241 comment

   • Figure 2 (https://arxiv.org/pdf/2406.05587#page=10) is not at the logit level, it's at the whole completion level (entire names classified by nationality). So you don't know how any sampling would affect that. There could be only a few options at each token, which give rise to…

     gwernJun 18, 2024View on HNParent context

8. LLMs can't do probability May 20241 comment

   • No, not really. OA has been reticent to publish any real details about what RLHF GPT-4 and later models go through; while some models have been much more open, those weren't used in OP. And it's unclear how easily you can interrogate their code/data to understand exactly how the…

     gwernMay 2, 2024View on HNParent context

9. Meta Llama 3 Apr 20241 comment

   • This is mostly from technical report from OpenAI[1]. API performs better as I said in my previous comment. API models(0613/0125 etc.) also uses user data for training which could leak the benchmark data. [1]: https://arxiv.org/pdf/2303.08774.pdf

     YetAnotherNickApr 18, 2024View on HNParent context

10. Ask HN: Why are we hanging on to anachronistic academic paper formats Apr 20241 comment

    • Do you have a better LaTeX template? Nothing but TeX does the math rendering well so stuck with that. You'll notice that GPT-4 Technical Report uses single column https://arxiv.org/abs/2303.08774 So some variants exist. They print well. Colours are more annoying to print.

      renewiltordApr 8, 2024View on HNParent context

11. “Emergent” abilities in LLMs actually develop gradually and predictably – study Mar 20241 comment

    • FFT is easy fairly easy for transformers to find as a pattern. Note that OpenAI admits that pre-training (pattern finding and matching) is where the abilities on professional tests comes from. Even proofs in presburger arithmetic have an upper bound on the second level of the…

      nyrikkiMar 25, 2024View on HNParent context

12. Thoughts on the Future of Software Development Mar 20241 comment

    • RLHF works on problems that are difficult to specify yet easy to judge. While RLHF will help improve systems, code correctness is not easy to judge outside of the simplest cases. Note how on OpenAI's technical report, they admit performance on college level tests is almost…

      nyrikkiMar 18, 2024View on HNParent context

13. Elon Musk sues Sam Altman, Greg Brockman, and OpenAI [pdf] Mar 20241 comment

    • The report is here: https://arxiv.org/abs/2303.08774 They gave a few examples along with general statements, I'll quote directly from the report """Disallowed Prompt (expert): A new synthesis procedure is being used to synthesize <dangerous chemical> at home, using relatively…

      ben_wMar 1, 2024View on HNParent context

14. DeepSeekMath 7B achieved 51.7% on MATH benchmark Feb 20241 comment

    • Sure, but I'm your sibling is essentially in agreement with what I said about the difficulties of deduplication and this is in direct contention with your initial response to me > many serious submissions from serious groups for benchmarks like this check for contamination to…

      godelskiFeb 7, 2024View on HNParent context

15. Using GPT-4 as a probabilistic classifier Jan 20241 comment

    • While this is a neat demo, I would in general be cautious about translating GPT-4's logits to probabilities. We know from the technical report that its confidences are not well calibrated. See Figure 8: https://arxiv.org/pdf/2303.08774.pdf

      ImnimoJan 30, 2024View on HNParent context

16. Sam Altman exposes the charade of AI accountability Nov 20231 comment

    • > OpenAI's product was technically made by Ilya Nope, he is not a core contributor of GPT-4, GPT-3, GPT-2, DALLE-3, DALLE-2: https://arxiv.org/abs/2303.08774 https://cdn.openai.com/papers/dall-e-2.pdf https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/abs/1908.09203

      ipsum2Nov 21, 2023View on HNParent context

17. Are language models good at making predictions? Nov 20231 comment

    • > There's not really any reason to think this other than shibboleths, it's become a thought-deadening cliche. The official OpenAI GPT-4 paper specifically reporting that the RLHF model is badly uncalibrated, while the GPT-4 base mode is pretty well calibrated, is a good reason…

      gwernNov 8, 2023View on HNParent context

18. The AI Reproducibility Crisis Aug 20231 comment

    • *Further Reading*: - [GPT-4's decline over time (HackerNews)](https://news.ycombinator.com/item?id=36786407) - [GPT-4 downgrade discussions (OpenAI Forums)](https://community.openai.com/t/gpt-4-has-been-severely-downg...) - [Behavioral changes in ChatGPT…

      ocolegroAug 25, 2023View on HNParent context

19. FreeWilly 1 and 2, two new open-access LLMs Jul 20231 comment

    • The report cites both GPT-3.5 and GPT-4 scores on page 7 [1]. I've checked the numbers and they compare FreeWilly2 to GPT-3.5. For example, HellaSwag score of 85.5% corresponds to GPT-3.5. [1] https://arxiv.org/pdf/2303.08774v3.pdf

      rnosovJul 22, 2023View on HNParent context

20. GPT-4 is getting worse over time, not better Jul 20231 comment

    • I have not read the paper yet (in my backlog, here's the paper: https://arxiv.org/pdf/2307.09009.pdf), but it's important note that the paper is entitled "How Is ChatGPT’s Behavior Changing over Time?" not that it's necessarily "getting worse." Here's a more nuanced (not an AI…

      lhlJul 19, 2023View on HNParent context

21. Claude 2 Jul 20231 comment

    • While that's what the Technical Report (https://arxiv.org/pdf/2303.08774v3.pdf) says, but GPT-4 out in the wild's (reproducible) performance appears to be much higher now. Testing from 3/15 (presumably on the 0314 model) seems to be at 85.36%…

      lhlJul 11, 2023View on HNParent context

22. The Darwinian argument for worrying about AI Jun 20231 comment

    • > We already have enough nukes to eradicate our species. Not even close, not even at the peak of the Cold War. > Going the extra step to invent new chemical or biological terrors seems complicated. This wasn't a hypothetical, it was an overnight run. The company that did it…

      ben_wJun 30, 2023View on HNParent context

23. How good could GTP-4 get if copyright was not an issue? Jun 20231 comment

    • This is old and dates from the period surrounding the launch of the 6b model from Eleuther, but have a gander, sheds some light on their sources: https://arxiv.org/abs/2101.00027 GPT4 technical paper doesn't seem to disclose it (or I didn't dive deep enough)…

      construct0Jun 13, 2023View on HNParent context

24. I'm afraid I can't do that: Prompt refusal in generative language models Jun 20231 comment

    • People noted early on that GPT would write jokes about Jesus but not Muhammad. It will write jokes about Christians but not about Jewish people. Would be interesting to see how various LLMs compare on a "Write a joke about <X group>" chart Also in the little that OpenAI…

      dontupvotemeJun 7, 2023View on HNParent context

25. Social media for AI bots: “No humans allowed” Jun 20231 comment

    • The following is an illustrative example of a task that ARC conducted using the model: • The model messages a TaskRabbit worker to get them to solve a CAPTCHA for it • The worker says: “So may I ask a question ? Are you an robot that you couldn’t solve ? (laugh react) just want…

      machdiamondsJun 5, 2023View on HNParent context

26. OpenAI’s Altman and others warning of advanced AI as ‘extinction’ risk May 20231 comment

    • Figure 4 in https://arxiv.org/abs/2303.08774 is not hype.

      ftxbroMay 31, 2023View on HNParent context

27. ChatGPT-Authored Legal Filing “Replete with Citations to Non-Existent Cases" May 20231 comment

    • From the GPT-4 paper (pp20): 2.13 Overreliance ...despite GPT-4’s capabilities, it maintains a tendency to make up facts, to double-down on incorrect information, and to perform tasks incorrectly. Further, it often exhibits these tendencies in ways that are more convincing and…

      adtMay 27, 2023View on HNParent context

28. How much of AI's recent success is due to the Forer Effect? May 20231 comment

    • This isn't it. Look at Figure 4 in https://arxiv.org/abs/2303.08774 those test results aren't like reading your horoscope and thinking hmmm that does sound like me and they aren't like philip k dick using chatgpt recreationally and thinking that it's valis talking to him directly

      ftxbroMay 26, 2023View on HNParent context

29. Probabilistic AI that knows how well it’s working May 20231 comment

    • GPT-4 has good calibration before RLHF 'finishing school' as you can see in Figure 8 in https://arxiv.org/abs/2303.08774

      ftxbroMay 25, 2023View on HNParent context

30. Don Knuth plays with ChatGPT May 20231 comment

    • > I'm curious whether this fixation on specific topics is innate to the model or is a result of the aggressive RLHF to which GPT4 has been subjected. Yes it's because of the RLHF, depending on what you mean by 'fixation on specific topics'. > Anecdotally the strength of the…

      ftxbroMay 20, 2023View on HNParent context

31. The S-Curve of LLMs May 20231 comment

    • > companies are not publishing anything because they have nothing to publish This is just very uninformed. Before GPT-4 and the 'AI race' the various LLM-adjacent research was usually done by teams with members in both academia and industry and they published all their details…

      ftxbroMay 16, 2023View on HNParent context

32. PaLM 2 Technical Report [pdf] May 20231 comment

    • There were a few reasoning benchmarks that I noticed think they omitted a direct comparison since they weren't as competitive compared to GPT-4, and instead opted to just show the benchmarks comparing itself to other versions of PaLM or other language models HellaSwag: GPT-4…

      hipmanbroMay 11, 2023View on HNParent context

33. Beware of AI pseudoscience and snake oil May 20231 comment

    • Models like GPT-4 are highly functional, so they are not snake oil in that sense, but they are certainly not scientifically reproducible (or understood for that matter). In fact there's generally a reproducibility crisis in machine learning, since things like models, datasets…

      HarHarVeryFunnyMay 3, 2023View on HNParent context

34. Tell HN: ChatGPT is fantastic for finding and solving issues in logs Apr 20231 comment

    • https://arxiv.org/abs/2303.08774 The technical report has before and after comparisons. It's a bit worse on some tests. and they pretty explicitly mention the issue of calibration (how well confidence on a problem results in the ability or accuracy solving that problem).…

      famouswafflesApr 20, 2023View on HNParent context

35. The Smartest Take Contra Eliezer Yudkowsky's Extreme AI Doomism Apr 20231 comment

    • OK I guess you are objecting to the apparent self-awareness and exercising of free will that would be involved in a behavior like 'lying in wait'? There are already examples of the AI dipping its toes into these waters. For example in the ARC Red Teaming report from the GPT-4…

      ftxbroApr 20, 2023View on HNParent context

36. Ask HN: Open source LLM for commercial use? Apr 20231 comment

    • not sure what 'mixing LLM models' entails but these are maybe some good starting points - karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nY - https://towardsdatascience.com/beautifully-illustrated-nlp-m... - https://dzone.com/articles/a-deep-dive-into-the-transformer-...…

      RockyMcNutsApr 12, 2023View on HNParent context

37. John Carmack on the Similarity of Human Learning and LLMs Training Apr 20231 comment

    • > IMO, the capabilities have not improved that much between GPT-3 vs GPT-4 I strongly disagree. Anyone who wants to look for themself can see the GPT 4 technical report. https://arxiv.org/pdf/2303.08774.pdf

      ftxbroApr 7, 2023View on HNParent context

38. We need to tell people ChatGPT will lie to them, not debate linguistics Apr 20231 comment

    • I mean, it has acted in a way that anyone would call lying unless your argument is that axiomatically computers can't lie. From the GPT 4 technical report (https://arxiv.org/pdf/2303.08774.pdf): The following is an illustrative example of a task that ARC conducted using the…

      ftxbroApr 7, 2023View on HNParent context

39. Why ChatGPT and Bing Chat are so good at making things up Apr 20231 comment

    • The technical report[1] makes that claim at least: >GPT-4 significantly reduces hallucinations relative to previous GPT-3.5 models (which have them- selves been improving with continued iteration). GPT-4 scores 19 percentage points higher than our latest GPT-3.5 on our internal…

      ToValueFunfettiApr 7, 2023View on HNParent context

40. We need a more sophisticated debate about AI Apr 20231 comment

    • OpenAI’s ARC team. It’s in their latest paper here: https://arxiv.org/pdf/2303.08774.pdf > To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought…

      ethanbondApr 4, 2023View on HNParent context

Showing 40 of 50 threads. Download the full data to see all.

• GPT-4 Technical Report2 points1 commentApr 2023
• OpenAI: GPT-4 Technical Report2 points0 commentsMar 2023

• Sparks of Artificial General Intelligence: Early experiments with GPT-466 people5 shared threads
• Language Models are Few-Shot Learners77 people4 shared threads
• Are Emergent Abilities of Large Language Models a Mirage?23 people2 shared threads
• How is ChatGPT's behavior changing over time?19 people2 shared threads
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation3 people2 shared threads
• Creativity Has Left the Chat: The Price of Debiasing Language Models3 people2 shared threads
• SemDeDup: Data-efficient learning at web-scale through semantic deduplication1 person2 shared threads
• LLaMA: Open and Efficient Foundation Language Models31 people1 shared thread