Umeå University's logo

umu.sePublikasjoner
Endre søk
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Sim-to-real transfer of active suspension control using deep reinforcement learning
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för fysik.
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för fysik.
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för fysik. (Digital fysik)
Skogforsk (the Forestry Research Institute of Sweden), Uppsala, Sweden.
Vise andre og tillknytning
(engelsk)Manuskript (preprint) (Annet vitenskapelig)
Abstract [en]

We explore sim-to-real transfer of deep reinforcement learning controllers for a heavy vehicle with active suspensions designed for traversing rough terrain. While related research primarily focuses on lightweight robots with electric motors and fast actuation, this study uses a forestry vehicle with a complex hydraulic driveline and slow actuation. We simulate the vehicle using multibody dynamics and apply system identification to find an appropriate set of simulation parameters. We then train policies in simulation using various techniques to mitigate the sim-to-real gap, including domain randomization, action delays, and a reward penalty to encourage smooth control. In reality, the policies trained with action delays and a penalty for erratic actions perform at nearly the same level as in simulation. In experiments on level ground, the motion trajectories closely overlap when turning to either side, as well as in a route tracking scenario. When faced with a ramp that requires active use of the suspensions, the simulated and real motions are in close alignment. This shows that the actuator model together with system identification yields a sufficiently accurate model of the actuators. We observe that policies trained without the additional action penalty exhibit fast switching or bang-bang control. These present smooth motions and high performance in simulation but transfer poorly to reality. We find that policies make marginal use of the local height map for perception, showing no indications of look-ahead planning. However, the strong transfer capabilities entail that further development concerning perception and performance can be largely confined to simulation. 

Emneord [en]
autonomous vehicles, rough terrain navigation, machine learning, sim-to-real, reinforcement learning, heavy vehicles
HSV kategori
Forskningsprogram
fysik; data- och systemvetenskap
Identifikatorer
URN: urn:nbn:se:umu:diva-207977DOI: 10.48550/arXiv.2306.11171OAI: oai:DiVA.org:umu-207977DiVA, id: diva2:1754988
Forskningsfinansiär
Mistra - The Swedish Foundation for Strategic Environmental Research, DIA 2017/14 #6Tilgjengelig fra: 2023-05-05 Laget: 2023-05-05 Sist oppdatert: 2023-07-04bibliografisk kontrollert
Inngår i avhandling
1. Terrain machine learning
Åpne denne publikasjonen i ny fane eller vindu >>Terrain machine learning
2023 (engelsk)Doktoravhandling, med artikler (Annet vitenskapelig)
Alternativ tittel[sv]
Maskininlärning i terräng
Abstract [en]

The use of heavy vehicles in rough terrain is vital in the industry but has negative implications for the climate and ecosystem. In addition, the demand for improved efficiency underscores the need to enhance these vehicles' navigation capabilities. Navigating rough terrain presents distinct challenges, including deformable soil, surface roughness, and spatial and temporal terrain variability. Focusing on forestry, this thesis aims to improve navigation using machine learning and physics simulations. Without considering the vehicle-terrain dynamics, methods for navigation can result in unsafe or unnecessarily challenging situations. Specifically, we address route planning, control for autonomous vehicles, and soilde formations. We simulate soil using the discrete element method and vehicles using multibody dynamics.

To enhance route planning, we train a predictor model that uses a height map of the terrain to predict measures of traversability. The model has a directional dependency, couples geometric terrain features with vehicle design and dynamics, and allows for swift evaluations over large areas. The proposed method facilitates detailed route planning, using multiple objectives to yield efficient solutions.

We address autonomy in rough terrain navigation by training a controller through deep reinforcement learning. The control policy uses a local height map for perception to plan and control a forwarder with actively articulated suspensions. The controller adapts to overcome various obstacles and demonstrates skilled driving in rough terrain.

Extending beyond simulation, we address the simulation-to-reality gap of vehicles with complex hydraulic drivelines through system identification and domain randomization. The results show that having an accurate model of the actuators, modelling system delays, and preventing bang-bang control yields successful transfer. Controllers that train in simulation and transfer to reality are a step toward autonomous vehicles.

While the previously mentioned studies assume rigid terrain, we also answer if the discrete element method can capture large soil deformations due to heavy traffic. The results show that the discrete element method can represent a wide variety of natural soil and that the resulting rut depths agree well with empirical models and experimental data, including multipass scenarios.

sted, utgiver, år, opplag, sider
Umeå: Umeå University, 2023. s. 38
Emneord
multibody dynamics simulation, rough terrain vehicle, autonomous vehicles, robotics control, discrete element method, sim-to-real, reinforcement learning
HSV kategori
Forskningsprogram
fysik
Identifikatorer
urn:nbn:se:umu:diva-207982 (URN)978-91-8070-060-3 (ISBN)978-91-8070-059-7 (ISBN)
Disputas
2023-06-01, NAT.D.410, Umeå, 09:00 (engelsk)
Opponent
Veileder
Forskningsfinansiär
Mistra - The Swedish Foundation for Strategic Environmental Research, DIA 2017/14 #6
Tilgjengelig fra: 2023-05-11 Laget: 2023-05-05 Sist oppdatert: 2023-05-08bibliografisk kontrollert

Open Access i DiVA

Fulltekst mangler i DiVA

Andre lenker

Forlagets fulltekstSim-to-real transfer of active suspension control using deep reinforcement learning

Person

Wiberg, ViktorWallin, ErikFälldin, ArvidWadbro, EddieServin, Martin

Søk i DiVA

Av forfatter/redaktør
Wiberg, ViktorWallin, ErikFälldin, ArvidWadbro, EddieServin, Martin
Av organisasjonen

Søk utenfor DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric

doi
urn-nbn
Totalt: 254 treff
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf