Umeå University's logo

umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Sim-to-real transfer of active suspension control using deep reinforcement learning
Umeå University, Faculty of Science and Technology, Department of Physics.
Umeå University, Faculty of Science and Technology, Department of Physics.
Umeå University, Faculty of Science and Technology, Department of Physics. (Digital fysik)
Skogforsk (the Forestry Research Institute of Sweden), Uppsala, Sweden.
Show others and affiliations
(English)Manuscript (preprint) (Other academic)
Abstract [en]

We explore sim-to-real transfer of deep reinforcement learning controllers for a heavy vehicle with active suspensions designed for traversing rough terrain. While related research primarily focuses on lightweight robots with electric motors and fast actuation, this study uses a forestry vehicle with a complex hydraulic driveline and slow actuation. We simulate the vehicle using multibody dynamics and apply system identification to find an appropriate set of simulation parameters. We then train policies in simulation using various techniques to mitigate the sim-to-real gap, including domain randomization, action delays, and a reward penalty to encourage smooth control. In reality, the policies trained with action delays and a penalty for erratic actions perform at nearly the same level as in simulation. In experiments on level ground, the motion trajectories closely overlap when turning to either side, as well as in a route tracking scenario. When faced with a ramp that requires active use of the suspensions, the simulated and real motions are in close alignment. This shows that the actuator model together with system identification yields a sufficiently accurate model of the actuators. We observe that policies trained without the additional action penalty exhibit fast switching or bang-bang control. These present smooth motions and high performance in simulation but transfer poorly to reality. We find that policies make marginal use of the local height map for perception, showing no indications of look-ahead planning. However, the strong transfer capabilities entail that further development concerning perception and performance can be largely confined to simulation. 

Keywords [en]
autonomous vehicles, rough terrain navigation, machine learning, sim-to-real, reinforcement learning, heavy vehicles
National Category
Computer graphics and computer vision Other Physics Topics
Research subject
Physics; computer and systems sciences
Identifiers
URN: urn:nbn:se:umu:diva-207977DOI: 10.48550/arXiv.2306.11171OAI: oai:DiVA.org:umu-207977DiVA, id: diva2:1754988
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research, DIA 2017/14 #6Available from: 2023-05-05 Created: 2023-05-05 Last updated: 2025-02-01Bibliographically approved
In thesis
1. Terrain machine learning
Open this publication in new window or tab >>Terrain machine learning
2023 (English)Doctoral thesis, comprehensive summary (Other academic)
Alternative title[sv]
Maskininlärning i terräng
Abstract [en]

The use of heavy vehicles in rough terrain is vital in the industry but has negative implications for the climate and ecosystem. In addition, the demand for improved efficiency underscores the need to enhance these vehicles' navigation capabilities. Navigating rough terrain presents distinct challenges, including deformable soil, surface roughness, and spatial and temporal terrain variability. Focusing on forestry, this thesis aims to improve navigation using machine learning and physics simulations. Without considering the vehicle-terrain dynamics, methods for navigation can result in unsafe or unnecessarily challenging situations. Specifically, we address route planning, control for autonomous vehicles, and soilde formations. We simulate soil using the discrete element method and vehicles using multibody dynamics.

To enhance route planning, we train a predictor model that uses a height map of the terrain to predict measures of traversability. The model has a directional dependency, couples geometric terrain features with vehicle design and dynamics, and allows for swift evaluations over large areas. The proposed method facilitates detailed route planning, using multiple objectives to yield efficient solutions.

We address autonomy in rough terrain navigation by training a controller through deep reinforcement learning. The control policy uses a local height map for perception to plan and control a forwarder with actively articulated suspensions. The controller adapts to overcome various obstacles and demonstrates skilled driving in rough terrain.

Extending beyond simulation, we address the simulation-to-reality gap of vehicles with complex hydraulic drivelines through system identification and domain randomization. The results show that having an accurate model of the actuators, modelling system delays, and preventing bang-bang control yields successful transfer. Controllers that train in simulation and transfer to reality are a step toward autonomous vehicles.

While the previously mentioned studies assume rigid terrain, we also answer if the discrete element method can capture large soil deformations due to heavy traffic. The results show that the discrete element method can represent a wide variety of natural soil and that the resulting rut depths agree well with empirical models and experimental data, including multipass scenarios.

Place, publisher, year, edition, pages
Umeå: Umeå University, 2023. p. 38
Keywords
multibody dynamics simulation, rough terrain vehicle, autonomous vehicles, robotics control, discrete element method, sim-to-real, reinforcement learning
National Category
Robotics and automation Other Physics Topics Forest Science Computer graphics and computer vision
Research subject
Physics
Identifiers
urn:nbn:se:umu:diva-207982 (URN)978-91-8070-060-3 (ISBN)978-91-8070-059-7 (ISBN)
Public defence
2023-06-01, NAT.D.410, Umeå, 09:00 (English)
Opponent
Supervisors
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research, DIA 2017/14 #6
Available from: 2023-05-11 Created: 2023-05-05 Last updated: 2025-02-05Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textSim-to-real transfer of active suspension control using deep reinforcement learning

Authority records

Wiberg, ViktorWallin, ErikFälldin, ArvidWadbro, EddieServin, Martin

Search in DiVA

By author/editor
Wiberg, ViktorWallin, ErikFälldin, ArvidWadbro, EddieServin, Martin
By organisation
Department of PhysicsDepartment of Computing Science
Computer graphics and computer visionOther Physics Topics

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 453 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf