HackerPapers

arXiv:1606.06565cs.AI

Concrete Problems in AI Safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané

Abstract

Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.

arXiv comments
29 pages
Journal reference
DOI
Subjects
cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 15 identifiable commenters in 16 comments across 15 threads, from Jun 2016 to Nov 2026. Found in 5 HN submissions from Jun 2016 to Oct 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments15 threads

  1. Astra can read and write weights to improve Qwen 1 comment

    • Why do you think the talk about AI safety is marketing as opposed to genuine fear? These are basically beliefs that the founders of the labs have had for a long time, since before LLMs were anywhere near as capable as they are today. For example, Dario has been publishing papers…

      reasonablekloutView on HNParent context

  2. Our position on open-weights models 1 comment

    • Ok but Dario has been thinking about AI Safety since 2016 [1], before even GPT-1. I think the simplest explanation is that the Anthropic folks genuinely believe what they say, it just happens to also help their business a lot. [1]: https://arxiv.org/abs/1606.06565

      reasonablekloutView on HNParent context

  3. Did Anthropic ask for this? 1 comment

    • Can you be more specific on what you mean - how would you prove what anyone believes about something short of reading minds and what would proof look like to you? Here is Dario writing about AI safety in 2016 [1]. Dario and others in the Anthropic circle have long been…

      ianm218View on HNParent context

  4. Yann LeCun and Andrew Ng: Why the 6-Month AI Pause Is a Bad Idea [video] 1 comment

    • The dangers of AGI I don't think are very related with consciousness or sentience (maybe the saddest outcome if it killed us all and weren't at all). I'm still catching up on my reading, but the existential risk of AGI as I understand it is chiefly that we don't know how to…

      lhlView on HNParent context

  5. Unsolved Problems in ML Safety 1 comment

    • Some overlap here with the 2017 paper from Google Brain et al: Concrete Problems in AI Safety: https://arxiv.org/pdf/1606.06565.pdf A nice video from the wonderful AI safety communicator Robert Miles outlines those problems: https://www.youtube.com/watch?v=AjyM-f8rDpg

      unholinessView on HNParent context

  6. Anthropic AI 1 comment

    • Their paper "Concrete problems in AI safety"[1] is interesting. Could be more concrete. They're run into the "common sense" problem, which I sometimes define, for robots, as "getting through the next 30 seconds without screwing up". They're trying to address it by playing with…

      AnimatsView on HNParent context

  7. SafeLife: AI Safety Environments Based on Conway's Game of Life 1 comment

    • Yes, this is one specific safety problem -- there are many other RL safety problems that deserve high quality benchmarks too. See eg https://arxiv.org/pdf/1606.06565.pdf or https://medium.com/@deepmindsafetyresearch/building-safe-art... for discussions of the problem space.

      pde3View on HNParent context

  8. Safety Gym 1 comment

    • I think the answer is that "just building in costs" is actually rather hard to get right. Check out how Concrete Problems in AI Safety (Section 6 in particular is about safe exploration) https://arxiv.org/pdf/1606.06565.pdf Quote: In practice, real world RL projects can often…

      Tyr42View on HNParent context

  9. Ask HN: Examples of AI ethical principles? 1 comment

    • I would add, as a compromise with the superintelligence true believers, "Concrete Problems in AI Safety" is quite good, and worth reading and thinking about. (It's especially relevant to RL systems that interact with the world.) https://arxiv.org/pdf/1606.06565v2.pdf

      currymjView on HNParent context

  10. OpenAI Research 1 comment

    • No, I wasn't aware of that. This is a great link, thanks! To anyone else here is their excellent 2016 paper linked from the above: "Concrete Problems in AI Safety" https://arxiv.org/abs/1606.06565 and direct link to 29-page PDF: https://arxiv.org/pdf/1606.06565.pdf [PDF!]…

      logicalleeView on HNParent context

  11. OpenAI's Researchers: Protecting Against AI’s Existential Threat 1 comment

    • This is Dario Amodei, head of the OpenAI safety team. We are devoting a substantial fraction of the organization’s bandwidth to safety, both on the side of technical research (where we have several people currently working on it full-time, and are very actively hiring for more…

      damodeiView on HNParent context

  12. The Last Invention of Man 1 comment

    • "but little or no scientific work to be done." Quite a lot has been written about what scientific work needs to be done. These papers try to summarize possible research directions: https://arxiv.org/pdf/1606.06565.pdf https://intelligence.org/files/TechnicalAgenda.pdf

      apsec112View on HNParent context

  13. The world’s biggest problems and why they’re not what first comes to mind 1 comment

    • There's also real research progress being made. Here are some of the issues people are working on: https://arxiv.org/abs/1606.06565

      BenjaminToddView on HNParent context

  14. Simple questions to help reduce AI hype 1 comment

    • According to this paper https://arxiv.org/pdf/1606.06565.pdf, OpenAI (John Schulman and Paul Christiano) explicitly indicate interest in researching transparency and strongly support work on fairness.

      evc123View on HNParent context

  15. Concrete AI Safety Problems 2 comments

    • Direct link to the paper: https://arxiv.org/pdf/1606.06565v1.pdf

      chrisfosterelliView on HNParent context

    • Paper: https://arxiv.org/pdf/1606.06565v1.pdf Google Post: https://research.googleblog.com/2016/06/bringing-precision-t... It was a pleasure for us to work on this with OpenAI and others. John/Paul/Jacob are good friends, and wonderful colleagues! :)

      colah3View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.