How is ChatGPT's behavior changing over time? · HackerPapers discussions

How is ChatGPT's behavior changing over time? — mentioned by vlyan

Mon, 28 Sep 2026 11:28:54 GMT

it's been known since the early days of GPT-4 that they alter the models whenever they feel like it. https://arxiv.org/pdf/2307.09009 the accusations are quite a few because people notice.

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by sporkland

Thu, 11 Jun 2026 22:52:12 GMT

Was hoping to understand if anyone has done any research on it. I could only really find one research paper on the topic [1], but it seems less focused on this specific issue. [1] https://arxiv.org/abs/2307.09009

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by b65e8bee43c2ed0

Fri, 17 Apr 2026 21:39:08 GMT

my source is that people have been noticing this since GPT4 days. https://arxiv.org/pdf/2307.09009 but of course, this isn't a written statement by a corporate spokespersyn. I don't think that breweries make such statements when they water their beer either.

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — HN submission

Sun, 11 Aug 2024 12:12:39 GMT

Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?

Has there been any consensus on this phenomenon I've seen where there are reports of decrease in model performance? This decrease in quality seems to take many forms: laziness, lower expressiveness, mistakes, laziness etc. One the other hand there are people who claim there hasn't been any degradation at all[0]? If there is indeed no degradation how could the perceived degradation be explained? [0] https://community.openai.com/t/declining-quality-of-openai-m... https://community.openai.com/t/declining-quality-of-openai-m... https://www.reddit.com/r/OpenAI/comments/18sc92o/with_all_th...…

42 points; 40 discussion replies on HN. Paper and source history. A submission is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by armchairhacker

Sat, 03 Aug 2024 05:00:50 GMT

At this point, someone should be periodically testing ChatGPT by scoring it against a set of questions (constant, not public, of varying difficulty, and enough that the score is consistent if you run the multiple times on the same day). If ChatGPT's score declines, there's clear evidence that OpenAI made it worse. Otherwise, I really doubt any of these reports. "ChatGPT got worse" is so common that, if it was true, there'd be solid emperical evidence (there's the Stanford study, https://arxiv.org/pdf/2307.09009, which shows ChatGPT can degrade across releases, and I believe that; but these…

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by BoorishBears

Wed, 27 Mar 2024 17:59:47 GMT

Databricks is trying to go all-in on convincing organizations they need to use in-house models, and therefore pay they to provide LLMOps. They're so far into this that their CTO co-authored a borderline dishonest study which got a ton of traction last summer trying to discredit GPT-4: https://arxiv.org/pdf/2307.09009.pdf

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by gwern

Mon, 26 Feb 2024 18:47:57 GMT

> For better or worse, the quality of output LLMs can produce based on such publicly available data appears to be peaking. Recent work has documented that GPT, for example, has begun to exhibit declines in the quality of its output.3 It suggests no such thing, and the problems with citation 3 (https://arxiv.org/abs/2307.09009) have been extensively discussed. > It has been suggested this decline in performance is the expected result of a feedback loop, wherein data collected for training is increasingly contaminated by GPT itself, as users leverage the technology to produce and post content…

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by capableweb

Tue, 05 Sep 2023 15:59:04 GMT

It is on my internet! https://arxiv.org/pdf/2307.09009.pdf

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by NikolaNovak

Tue, 05 Sep 2023 15:33:38 GMT

* had to shorten the title to fit HN size. No editorializing intended * original study: https://arxiv.org/pdf/2307.09009.pdf * thought it's interesting as we hear a lot of people's experience, but this seems structured and well tracked set of specific scenarios

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by modeless

Sat, 26 Aug 2023 06:35:45 GMT

You are wrong. What I described is exactly what they did in the "math" benchmark of v1 of the study (https://arxiv.org/pdf/2307.09009v1.pdf). They asked "is this number prime" for a bunch of prime numbers. The old version gave a bunch of "reasoning" that was actually faulty and then guessed "yes". The new version guessed "no" (which is arguably a better guess as to whether a random number is prime). In neither case did it actually do the work required to answer the question and the change in "correct" answers is an illusion. In the programming category the newer GPT-4 actually performed…

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by esperent

Sat, 26 Aug 2023 04:37:53 GMT

No it's not. Here's a link to the study, you can check the questions they asked. https://arxiv.org/pdf/2307.09009.pdf

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by ocolegro

Fri, 25 Aug 2023 19:25:45 GMT

*Further Reading*: - [GPT-4's decline over time (HackerNews)](https://news.ycombinator.com/item?id=36786407) - [GPT-4 downgrade discussions (OpenAI Forums)](https://community.openai.com/t/gpt-4-has-been-severely-downg...) - [Behavioral changes in ChatGPT (arXiv)](https://arxiv.org/abs/2307.09009) - [Zero-Shot Replication Effort (Github)](https://github.com/emrgnt-cmplxty/zero-shot-replication) - [Inconsistencies in GPT-4 HumanEval (Github)](https://github.com/evalplus/evalplus/issues/15) - [Early experiments with GPT-4 (arXiv)](https://arxiv.org/abs/2303.12712) - [GPT-4 Technical Report…

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — HN submission

Fri, 25 Aug 2023 19:15:29 GMT

The AI Reproducibility Crisis

I've really been struggling of late to replicate recent findings in research that has built on top of GPT-3.5/GPT-4. This leads me to believe there is growing yet largely unnoticed issue is taking root in recent AI research. I've termed this the "AI Reproducibility Crisis". The principle is simple, if accessible private models are silently changing in time, then previous results cannot be replicated. *Key Issues*: 1. Users have reported significant performance shifts post the May release. 2. Beyond community discussions, academic studies are showing differences in performance across time…

4 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by shubb

Mon, 14 Aug 2023 00:28:43 GMT

Thank you for pointing that out - I had assumed that things were not how they are. Although performance has varied over time https://arxiv.org/pdf/2307.09009.pdf I also notice that the API allows you to use a frozen version of the model which avoids the worries I mentioned.

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by n2d4

Thu, 10 Aug 2023 13:49:38 GMT

This one [1]? That tracks two distinct versions (0613 vs. 0314). Also, IMO, the tasks they evaluate aren't useful (I rarely want my LLM to tell me whether 17077 is a prime number), and there's room for cherrypicking/survivorship bias. My guess is that OpenAI did something between 0314 and 0613 that shifted focus away from maths to other subjects. [1] https://arxiv.org/pdf/2307.09009.pdf

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by ndr_

Fri, 04 Aug 2023 18:06:39 GMT

This article is rooted in the junk science by James Zou et al, published in their paper "How is ChatGPT's behavior changing over time?" (https://arxiv.org/abs/2307.09009). Initially, one of their benchmark scripts stumbled upon backticks added by GPT-4, causing poor scores. Simon Boehm fixed it for them, scores improved significantly (https://twitter.com/Si_Boehm/status/1681801371656536068). Now the authors are back with a revision (without giving credit to Boehm). Still, the benchmarks are silly because they measure what LLMs are historically challenged with: math and cases of "Do not do x"…

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by isaacfrond

Wed, 02 Aug 2023 21:04:58 GMT

Here’s a study showing how chatGpt’s math deteriorated. https://arxiv.org/pdf/2307.09009.pdf

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by WithinReason

Wed, 02 Aug 2023 08:50:46 GMT

It is and it's documented: https://arxiv.org/abs/2307.09009

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by isaacfrond

Sun, 23 Jul 2023 09:10:49 GMT

research paper: https://arxiv.org/pdf/2307.09009.pdf

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by embeng4096

Fri, 21 Jul 2023 15:28:45 GMT

re: GPT4 degrading -- From this recent study: https://arxiv.org/pdf/2307.09009.pdf > For example, GPT-4 (March 2023) was very good at identifying prime numbers (accuracy 97.6%) but GPT-4 (June 2023) was very poor on these same questions (accuracy 2.4%). There is more in the study but I just grabbed a short snippet as an example.

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by SmartVA

Fri, 21 Jul 2023 01:45:51 GMT

Study: https://arxiv.org/pdf/2307.09009.pdf

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — HN submission

Thu, 20 Jul 2023 15:30:02 GMT

How is ChatGPT's behavior changing over time?

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

How is ChatGPT's behavior changing over time? — mentioned by lhl

Wed, 19 Jul 2023 14:21:28 GMT

I have not read the paper yet (in my backlog, here's the paper: https://arxiv.org/pdf/2307.09009.pdf), but it's important note that the paper is entitled "How Is ChatGPT’s Behavior Changing over Time?" not that it's necessarily "getting worse." Here's a more nuanced (not an AI clout chasing account) discussion by Arvind Narayanan (Princeton CS prof) about the results: https://twitter.com/random_walker/status/1681489529494970368 One thing that I have confirmed is while the abstract and intro talk about evaluating "code generation" as if GPT-4 code generation is getting worse, In is 3.3/Figure…

Paper and discussion history. A mention is not an endorsement.

How is ChatGPT's behavior changing over time? — HN submission

Wed, 19 Jul 2023 01:58:33 GMT

How is ChatGPT's behavior changing over time?

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

How is ChatGPT's behavior changing over time? — HN submission

Wed, 19 Jul 2023 01:06:12 GMT

How is ChatGPT's behavior changing over time?

289 points; 178 discussion replies on HN. Paper and source history. A submission is not an endorsement.