Umeå universitets logga

umu.sePublikationer
Ändra sökning
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Sim-to-real transfer of active suspension control using deep reinforcement learning
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för fysik.
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för fysik.
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för fysik. (Digital fysik)
Skogforsk (the Forestry Research Institute of Sweden), Uppsala, Sweden.
Visa övriga samt affilieringar
(Engelska)Manuskript (preprint) (Övrigt vetenskapligt)
Abstract [en]

We explore sim-to-real transfer of deep reinforcement learning controllers for a heavy vehicle with active suspensions designed for traversing rough terrain. While related research primarily focuses on lightweight robots with electric motors and fast actuation, this study uses a forestry vehicle with a complex hydraulic driveline and slow actuation. We simulate the vehicle using multibody dynamics and apply system identification to find an appropriate set of simulation parameters. We then train policies in simulation using various techniques to mitigate the sim-to-real gap, including domain randomization, action delays, and a reward penalty to encourage smooth control. In reality, the policies trained with action delays and a penalty for erratic actions perform at nearly the same level as in simulation. In experiments on level ground, the motion trajectories closely overlap when turning to either side, as well as in a route tracking scenario. When faced with a ramp that requires active use of the suspensions, the simulated and real motions are in close alignment. This shows that the actuator model together with system identification yields a sufficiently accurate model of the actuators. We observe that policies trained without the additional action penalty exhibit fast switching or bang-bang control. These present smooth motions and high performance in simulation but transfer poorly to reality. We find that policies make marginal use of the local height map for perception, showing no indications of look-ahead planning. However, the strong transfer capabilities entail that further development concerning perception and performance can be largely confined to simulation. 

Nyckelord [en]
autonomous vehicles, rough terrain navigation, machine learning, sim-to-real, reinforcement learning, heavy vehicles
Nationell ämneskategori
Datorseende och robotik (autonoma system) Annan fysik
Forskningsämne
fysik; data- och systemvetenskap
Identifikatorer
URN: urn:nbn:se:umu:diva-207977DOI: 10.48550/arXiv.2306.11171OAI: oai:DiVA.org:umu-207977DiVA, id: diva2:1754988
Forskningsfinansiär
Mistra - Stiftelsen för miljöstrategisk forskning, DIA 2017/14 #6Tillgänglig från: 2023-05-05 Skapad: 2023-05-05 Senast uppdaterad: 2023-07-04Bibliografiskt granskad
Ingår i avhandling
1. Terrain machine learning
Öppna denna publikation i ny flik eller fönster >>Terrain machine learning
2023 (Engelska)Doktorsavhandling, sammanläggning (Övrigt vetenskapligt)
Alternativ titel[sv]
Maskininlärning i terräng
Abstract [en]

The use of heavy vehicles in rough terrain is vital in the industry but has negative implications for the climate and ecosystem. In addition, the demand for improved efficiency underscores the need to enhance these vehicles' navigation capabilities. Navigating rough terrain presents distinct challenges, including deformable soil, surface roughness, and spatial and temporal terrain variability. Focusing on forestry, this thesis aims to improve navigation using machine learning and physics simulations. Without considering the vehicle-terrain dynamics, methods for navigation can result in unsafe or unnecessarily challenging situations. Specifically, we address route planning, control for autonomous vehicles, and soilde formations. We simulate soil using the discrete element method and vehicles using multibody dynamics.

To enhance route planning, we train a predictor model that uses a height map of the terrain to predict measures of traversability. The model has a directional dependency, couples geometric terrain features with vehicle design and dynamics, and allows for swift evaluations over large areas. The proposed method facilitates detailed route planning, using multiple objectives to yield efficient solutions.

We address autonomy in rough terrain navigation by training a controller through deep reinforcement learning. The control policy uses a local height map for perception to plan and control a forwarder with actively articulated suspensions. The controller adapts to overcome various obstacles and demonstrates skilled driving in rough terrain.

Extending beyond simulation, we address the simulation-to-reality gap of vehicles with complex hydraulic drivelines through system identification and domain randomization. The results show that having an accurate model of the actuators, modelling system delays, and preventing bang-bang control yields successful transfer. Controllers that train in simulation and transfer to reality are a step toward autonomous vehicles.

While the previously mentioned studies assume rigid terrain, we also answer if the discrete element method can capture large soil deformations due to heavy traffic. The results show that the discrete element method can represent a wide variety of natural soil and that the resulting rut depths agree well with empirical models and experimental data, including multipass scenarios.

Ort, förlag, år, upplaga, sidor
Umeå: Umeå University, 2023. s. 38
Nyckelord
multibody dynamics simulation, rough terrain vehicle, autonomous vehicles, robotics control, discrete element method, sim-to-real, reinforcement learning
Nationell ämneskategori
Robotteknik och automation Annan fysik Skogsvetenskap Datorseende och robotik (autonoma system)
Forskningsämne
fysik
Identifikatorer
urn:nbn:se:umu:diva-207982 (URN)978-91-8070-060-3 (ISBN)978-91-8070-059-7 (ISBN)
Disputation
2023-06-01, NAT.D.410, Umeå, 09:00 (Engelska)
Opponent
Handledare
Forskningsfinansiär
Mistra - Stiftelsen för miljöstrategisk forskning, DIA 2017/14 #6
Tillgänglig från: 2023-05-11 Skapad: 2023-05-05 Senast uppdaterad: 2023-05-08Bibliografiskt granskad

Open Access i DiVA

Fulltext saknas i DiVA

Övriga länkar

Förlagets fulltextSim-to-real transfer of active suspension control using deep reinforcement learning

Person

Wiberg, ViktorWallin, ErikFälldin, ArvidWadbro, EddieServin, Martin

Sök vidare i DiVA

Av författaren/redaktören
Wiberg, ViktorWallin, ErikFälldin, ArvidWadbro, EddieServin, Martin
Av organisationen
Institutionen för fysikInstitutionen för datavetenskap
Datorseende och robotik (autonoma system)Annan fysik

Sök vidare utanför DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetricpoäng

doi
urn-nbn
Totalt: 329 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf