Concrete Problems in AI Safety · HackerPapers discussions

Concrete Problems in AI Safety — mentioned by reasonableklout

Tue, 03 Nov 2026 18:31:46 GMT

Why do you think the talk about AI safety is marketing as opposed to genuine fear? These are basically beliefs that the founders of the labs have had for a long time, since before LLMs were anywhere near as capable as they are today. For example, Dario has been publishing papers on AI safety since 2016, before GPT-1 or even transformers existed: https://arxiv.org/abs/1606.06565 > so are anthropic and google, apparently Yes, because every company in this industry is rushing to build these models as fast as they can. They are taking these risks because they believe if they do not, their…

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — HN submission

Sun, 11 Oct 2026 22:46:36 GMT

Concrete Problems in AI Safety

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Concrete Problems in AI Safety — mentioned by reasonableklout

Mon, 27 Jul 2026 23:52:56 GMT

Ok but Dario has been thinking about AI Safety since 2016 [1], before even GPT-1. I think the simplest explanation is that the Anthropic folks genuinely believe what they say, it just happens to also help their business a lot. [1]: https://arxiv.org/abs/1606.06565

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by ianm218

Mon, 15 Jun 2026 00:26:46 GMT

Can you be more specific on what you mean - how would you prove what anyone believes about something short of reading minds and what would proof look like to you? Here is Dario writing about AI safety in 2016 [1]. Dario and others in the Anthropic circle have long been associated [2] with the effective altruism movement, which whatever you think about them, they are very concerned about AI existential risks. Ronan Farrow & the New Yorker did a mega deep dive on Sam Altmans history of being dishonest and manipulative, and it credibly reports that Anthropic cofounders left due to not trusting…

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by lhl

Fri, 07 Apr 2023 18:10:00 GMT

The dangers of AGI I don't think are very related with consciousness or sentience (maybe the saddest outcome if it killed us all and weren't at all). I'm still catching up on my reading, but the existential risk of AGI as I understand it is chiefly that we don't know how to align AI (and keep it aligned) to be beneficial/benevolent for humanity, but the risk is contingent also on a few other things things - that we develop AI that agentic and able to develop and pursue it's own goals and sub-goals, and that it has power-seeking or game-theory strategic awareness/planning. The extra dangerous…

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by unholiness

Mon, 11 Oct 2021 15:33:24 GMT

Some overlap here with the 2017 paper from Google Brain et al: Concrete Problems in AI Safety: https://arxiv.org/pdf/1606.06565.pdf A nice video from the wonderful AI safety communicator Robert Miles outlines those problems: https://www.youtube.com/watch?v=AjyM-f8rDpg

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by Animats

Fri, 28 May 2021 22:46:53 GMT

Their paper "Concrete problems in AI safety"[1] is interesting. Could be more concrete. They're run into the "common sense" problem, which I sometimes define, for robots, as "getting through the next 30 seconds without screwing up". They're trying to address it by playing with the weighting in goal functions for machine learning. They write "Yet intuitively it seems like it should often be possible to predict which actions are dangerous and explore in a way that avoids them, even when we don’t have that much information about the environment." For humans, yes. None of the tweaks on machine…

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by pde3

Thu, 05 Dec 2019 10:35:44 GMT

Yes, this is one specific safety problem -- there are many other RL safety problems that deserve high quality benchmarks too. See eg https://arxiv.org/pdf/1606.06565.pdf or https://medium.com/@deepmindsafetyresearch/building-safe-art... for discussions of the problem space.

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by Tyr42

Fri, 22 Nov 2019 04:22:16 GMT

I think the answer is that "just building in costs" is actually rather hard to get right. Check out how Concrete Problems in AI Safety (Section 6 in particular is about safe exploration) https://arxiv.org/pdf/1606.06565.pdf Quote: In practice, real world RL projects can often avoid these issues by simply hard-coding an avoidance of catastrophic behaviors. For instance, an RL-based robot helicopter might be programmed to override its policy with a hard-coded collision avoidance sequence (such as spinning its propellers to gain altitude) whenever it’s too close to the ground. This approach…

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by currymj

Sun, 07 Oct 2018 15:43:05 GMT

I would add, as a compromise with the superintelligence true believers, "Concrete Problems in AI Safety" is quite good, and worth reading and thinking about. (It's especially relevant to RL systems that interact with the world.) https://arxiv.org/pdf/1606.06565v2.pdf

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by logicallee

Tue, 15 May 2018 13:40:22 GMT

No, I wasn't aware of that. This is a great link, thanks! To anyone else here is their excellent 2016 paper linked from the above: "Concrete Problems in AI Safety" https://arxiv.org/abs/1606.06565 and direct link to 29-page PDF: https://arxiv.org/pdf/1606.06565.pdf [PDF!] However, that PDF does not mention "AGI" even a single time, except in reference 17 "The AGI containment problem". That is actually also avaialable online here - https://arxiv.org/abs/1604.00545 but doesn't seem to be what they have in mind. So it seems the use of the term "safe" is actually much more narrow in AI literature…

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — HN submission

Wed, 29 Nov 2017 19:12:57 GMT

AI Safety: problem of accidents in machine learning systems

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Concrete Problems in AI Safety — mentioned by damodei

Wed, 18 Oct 2017 18:49:38 GMT

This is Dario Amodei, head of the OpenAI safety team. We are devoting a substantial fraction of the organization’s bandwidth to safety, both on the side of technical research (where we have several people currently working on it full-time, and are very actively hiring for more: https://openai.com/jobs/), and in terms of what we think the right policy actions are (several of us, including me, have been doing a lot of talking to people in government about what we can expect from AI and how the government can help). Beyond that it’s just a very central part of our strategy — it’s important for…

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by apsec112

Fri, 06 Oct 2017 23:40:59 GMT

"but little or no scientific work to be done." Quite a lot has been written about what scientific work needs to be done. These papers try to summarize possible research directions: https://arxiv.org/pdf/1606.06565.pdf https://intelligence.org/files/TechnicalAgenda.pdf

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by BenjaminTodd

Tue, 30 May 2017 22:26:56 GMT

There's also real research progress being made. Here are some of the issues people are working on: https://arxiv.org/abs/1606.06565

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by evc123

Sun, 10 Jul 2016 08:10:40 GMT

According to this paper https://arxiv.org/pdf/1606.06565.pdf, OpenAI (John Schulman and Paul Christiano) explicitly indicate interest in researching transparency and strongly support work on fairness.

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — HN submission

Mon, 04 Jul 2016 08:47:11 GMT

Concrete Problems in AI Safety

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Concrete Problems in AI Safety — HN submission

Thu, 30 Jun 2016 15:40:02 GMT

Concrete Problems in AI Safety

17 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Concrete Problems in AI Safety — HN submission

Tue, 28 Jun 2016 03:57:02 GMT

Concrete Problems in AI Safety

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Concrete Problems in AI Safety — mentioned by colah3

Wed, 22 Jun 2016 00:54:34 GMT

Paper: https://arxiv.org/pdf/1606.06565v1.pdf Google Post: https://research.googleblog.com/2016/06/bringing-precision-t... It was a pleasure for us to work on this with OpenAI and others. John/Paul/Jacob are good friends, and wonderful colleagues! :)

Paper and discussion history. A mention is not an endorsement.

Concrete Problems in AI Safety — mentioned by chrisfosterelli

Wed, 22 Jun 2016 00:51:26 GMT

Direct link to the paper: https://arxiv.org/pdf/1606.06565v1.pdf

Paper and discussion history. A mention is not an endorsement.