Umeå University's logo

umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Policy regularization for legible behavior
Umeå University, Faculty of Science and Technology, Department of Computing Science.ORCID iD: 0000-0001-5993-3292
Umeå University, Faculty of Science and Technology, Department of Computing Science.ORCID iD: 0000-0001-7242-2200
2023 (English)In: Neural Computing & Applications, ISSN 0941-0643, E-ISSN 1433-3058, Vol. 35, no 23, p. 16781-16790Article in journal (Refereed) Published
Abstract [en]

In this paper we propose a method to augment a Reinforcement Learning agent with legibility. This method is inspired by the literature in Explainable Planning and allows to regularize the agent’s policy after training, and without requiring to modify its learning algorithm. This is achieved by evaluating how the agent’s optimal policy may produce observations that would make an observer model to infer a wrong policy. In our formulation, the decision boundary introduced by legibility impacts the states in which the agent’s policy returns an action that is non-legible because having high likelihood also in other policies. In these cases, a trade-off between such action, and legible/sub-optimal action is made. We tested our method in a grid-world environment highlighting how legibility impacts the agent’s optimal policy, and gathered both quantitative and qualitative results. In addition, we discuss how the proposed regularization generalizes over methods functioning with goal-driven policies, because applicable to general policies of which goal-driven policies are a special case.

Place, publisher, year, edition, pages
Springer, 2023. Vol. 35, no 23, p. 16781-16790
Keywords [en]
Reinforcement Learning, Transparency, Interpretability, Legibility
National Category
Computer Sciences
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:umu:diva-192813DOI: 10.1007/s00521-022-07942-7ISI: 000875293700002Scopus ID: 2-s2.0-85140636891OAI: oai:DiVA.org:umu-192813DiVA, id: diva2:1641039
Note

Originally included in thesis in manuscript form.

Available from: 2022-02-28 Created: 2022-02-28 Last updated: 2023-12-05Bibliographically approved
In thesis
1. Expressing and recognizing intentions
Open this publication in new window or tab >>Expressing and recognizing intentions
2022 (English)Doctoral thesis, comprehensive summary (Other academic)
Alternative title[sv]
Uttrycka och känna igen avsikter
Abstract [en]

With the advancement of Artificial Intelligence, intelligent computer programs known as agents are coming increasingly close to the life of human beings. In an optimistic view, predictions tell us that agents will be governing many machines such as phones, robots and cars. Agents will play an important role in our daily life, and because of this, it is becoming more and more relevant to provide them the capacity of interacting with their human users without requiring prior expert training, bur rather supporting understanding through common sense reasoning. Striving towards this objective, one important aspect of agents design is intentionality, that relates to their capacity of understanding goals and plans of their users, and to make theirs understood. The ability to reason on goals and objectives of themselves and others is especially important if the agents are autonomous such as in autonomous robots, because enabling them to interact with other agents by relying on their sensor data and internal computations only, and not on explicit data provided by their designer. 

Intentionality imbues agents with additional capacities supporting cooperative activities: if a helpful agent recognize an intention, it could proactively actuate interventions in an helpful way, such as giving advice. Alternatively, whenever the agent detects that its user is not understanding its objective it could communicate the mismatching information. As an illustrative example, let's consider the case of an autonomous car with a passenger on the road to his office. After checking the online maps the car identifies a traffic jam ahead, and to avoid it, autonomously decides to change its path mid-way by taking a less populated road. The behavior of the car is quite intelligent, however, if left unexplained, the change of intention would leave the passenger wandering  what is happening: he was on the main road to the office and suddenly the car turned left. This would at minimum force him to question the car what's going on. Rather, a continuous degree of understanding can be maintained if the car intelligently detect such mismatch of intentions by computing its passenger expectations, and thus preemptively communicate the new selected paths whenever required. 

This seemingly simple process of communicating changes in the intention, looks simple but it is a quite difficult one. It requires to reason on what are the intentions of the user, and how and when they should be aligned with those of the car, either explicitly through a process of explanation, or implicitly through a behavior that is interpretable by human beings.  To support these capacities it is becoming apparent how intelligent agents should leverage how we commonly think about things, referred to as common sense reasoning. Common sense reasoning relates to how we form conjectures based on what we observe, and agents forming conjectures in the same way could be better collaborators rather than those reasoning in other ways. In this thesis we utilized an established model for common sense reasoning known as Theory of Mind, of which the thesis will contain a brief introduction. 

By leveraging Theory of Mind and classical agent architectures, this thesis provides an initial formulation of a novel computational architecture capable of aggregating multiple tasks from intentionality, such as intent recognition, informative communication of intention and interpretable behavior. We refer to this model as the Mirror Agent Model, because envisioning the agent as interacting with a mirrored copy of itself whenever supporting intentionality. Inside the model expressing and recognizing intentions are two faces of the same coin and represent a dual to each other. This represents a step forward towards the unification in a single framework of many tasks related to intentionality, that are at the moment considered mostly independently inside the literature.

The thesis will firstly provide introductory chapters on agents, intentions and theory of mind reasoning, followed by a chapter describing the developed computational models. This chapter conceptually aggregates many of the algorithms from the papers and aims at providing an initial formulation of the Mirror Agent Model. Finally, the thesis will conclude with a summary of contributions and concluding remarks.

Place, publisher, year, edition, pages
Umeå: Umeå University, 2022. p. 80
Series
Report / UMINF, ISSN 0348-0542 ; 22.07
Keywords
agent, model, plan, action, human-robot interaction, robot, mirror agent model, intention, recognition, interpretable behavior, artificial intelligence
National Category
Robotics Computer Systems
Research subject
human-computer interaction; Computer Science
Identifiers
urn:nbn:se:umu:diva-198631 (URN)978-91-7855-768-4 (ISBN)978-91-7855-767-7 (ISBN)
Public defence
2022-09-16, NAT.D.360, Naturvetarhuset, Umeå, 13:15 (English)
Opponent
Supervisors
Available from: 2022-08-26 Created: 2022-08-16 Last updated: 2022-08-22Bibliographically approved

Open Access in DiVA

fulltext(878 kB)46 downloads
File information
File name FULLTEXT03.pdfFile size 878 kBChecksum SHA-512
32416da23a982eb81fe84eb0a46bec5f3e869b2abd84a93ed25f4e359815e671619be4ab1a68d888a14ffcf343593bbac1e388d2d274e64e4bb77b62957691a0
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Persiani, MicheleHellström, Thomas

Search in DiVA

By author/editor
Persiani, MicheleHellström, Thomas
By organisation
Department of Computing Science
In the same journal
Neural Computing & Applications
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 107 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 318 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf