arXiv:1902.09469cs.AI
Embedded Agency
Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.
Abstract
Traditional models of rational action treat the agent as though it is cleanly separated from its environment, and can act on that environment from the outside. Such agents have a known functional relationship with their environment, can model their environment in every detail, and do not need to reason about themselves or their internal parts. We provide an informal survey of obstacles to formalizing good reasoning for agents embedded in their environment. Such agents must optimize an environment that is not of type "function"; they must rely on models that fit within the modeled environment; and they must reason about themselves as just another physical system, made of parts that can be modified and that can work at cross purposes.
It reminds me of this paper from MIRI a few years ago discussing models which treat themselves as an explicit part of the environment. I think it's a very productive approach - https://arxiv.org/abs/1902.09469
falcor84View on HNParent context