Umeå University's logo

umu.sePublications
Change search
Link to record
Permanent link

Direct link
Wallin, Erik
Publications (10 of 24) Show all publications
Lundbäck, M., Fälldin, A., Wallin, E. & Servin, M. (2024). Learning forwarder trafficability from real and synthetic data. In: IUFRO 2024: Detailed programme. Paper presented at IUFRO 2024 - XXVI IUFRO World Congress, Stockholm, Sweden, June 23-29, 2024. , Article ID T5.30.
Open this publication in new window or tab >>Learning forwarder trafficability from real and synthetic data
2024 (English)In: IUFRO 2024: Detailed programme, 2024, article id T5.30Conference paper, Oral presentation with published abstract (Other academic)
Abstract [en]

Forwarder trafficability is a function of terrain and vehicle properties. Predicting trafficability is vital for energy efficient planning- and operator-assisting systems, as well as for remote and autonomous driving. Inaccurate or insufficient information can lead to inefficient paths, excessive fuel usage, equipment wear, and soil damages. Training trafficability models require data in a quantity hard to collect solely from in-field experiments, especially considering the need for data from situations ranging from very easy to non-traversable.

To circumvent this problem, we perform in-field system identification for a forwarder in the Nordic cut-to-length system, to obtain a calibrated multi-body dynamics simulation model traversing firm but potentially rough and blocky terrain. By letting the real-world forward derdrive in very difficult terrain, the model is able to reflect a wide range of real conditions. The model is used in simulations, where collecting large amounts of data from a variety of situations is easy, cheap, and hazard free. Using this data, a deep neural network is trained to predict trafficability in terms of attainable driving speed, energy consumption, and machine wear.

The resulting predictor model uses laser scanned terrains to efficiently produce trafficability measures with high fidelity and accuracy, e.g., depending on the vehicle’s precise location, speed, heading, and weight. Trafficability on wet and weak soil is not addressed in this work. The predictor model is machine specific, but general enough for practical application in diverse terrain conditions. Our emphasis on energy consumption enables elaborate calculations of emissions, profoundly contributing to sustainable forest operations. Apart from the benefits from reduced emissions, the model can also be used to optimize extraction trail routing, which is a major contributor to the total extraction cost. Rough terrain trafficability is only part of an optima loute, but it has been neglected in previous research. We see big potential in combining our predictor model with existing route optimization methods to achieve a more complete result. By creating an open library of annotated machine data and code for preparing input terrain-data and running the trafficability model, we enable adoption of the results by others and application in existing and new software.

National Category
Computer graphics and computer vision
Identifiers
urn:nbn:se:umu:diva-227460 (URN)
Conference
IUFRO 2024 - XXVI IUFRO World Congress, Stockholm, Sweden, June 23-29, 2024
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research
Available from: 2024-06-27 Created: 2024-06-27 Last updated: 2025-02-07Bibliographically approved
Wallin, E., Wiberg, V. & Servin, M. (2024). Multi-log grasping using reinforcement learning and virtual visual servoing. Robotics, 13(1), Article ID 3.
Open this publication in new window or tab >>Multi-log grasping using reinforcement learning and virtual visual servoing
2024 (English)In: Robotics, E-ISSN 2218-6581, Vol. 13, no 1, article id 3Article in journal (Refereed) Published
Abstract [en]

We explore multi-log grasping using reinforcement learning and virtual visual servoing for automated forwarding in a simulated environment. Automation of forest processes is a major challenge, and many techniques regarding robot control pose different challenges due to the unstructured and harsh outdoor environment. Grasping multiple logs involves various problems of dynamics and path planning, where understanding the interaction between the grapple, logs, terrain, and obstacles requires visual information. To address these challenges, we separate image segmentation from crane control and utilise a virtual camera to provide an image stream from reconstructed 3D data. We use Cartesian control to simplify domain transfer to real-world applications. Because log piles are static, visual servoing using a 3D reconstruction of the pile and its surroundings is equivalent to using real camera data until the point of grasping. This relaxes the limits on computational resources and time for the challenge of image segmentation, and allows for data collection in situations where the log piles are not occluded. The disadvantage is the lack of information during grasping. We demonstrate that this problem is manageable and present an agent that is 95% successful in picking one or several logs from challenging piles of 2–5 logs.

Place, publisher, year, edition, pages
MDPI, 2024
Keywords
autonomous forwarding, visual servoing, virtual camera, reinforcement learning, multi-log grasping, Cartesian control
National Category
Robotics and automation Control Engineering Other Physics Topics Computer graphics and computer vision
Research subject
Physics; Automatic Control
Identifiers
urn:nbn:se:umu:diva-220402 (URN)10.3390/robotics13010003 (DOI)001151117200001 ()2-s2.0-85183350095 (Scopus ID)
Projects
Mistra Digital Forest
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research, Grant DIA 2017/14 #6
Available from: 2024-02-02 Created: 2024-02-02 Last updated: 2025-02-05Bibliographically approved
Fälldin, A., Häggström, C., Höök, C., Jönsson, P., Lindroos, O., Lundbäck, M., . . . Servin, M. (2024). Open data, models, and software for machine automation. In: IUFRO 2024: Detailed programme. Paper presented at IUFRO 2024 - XXVI IUFRO World Congress, Stockholm, Sweden, June 23-29, 2024. , Article ID T5.10.
Open this publication in new window or tab >>Open data, models, and software for machine automation
Show others...
2024 (English)In: IUFRO 2024: Detailed programme, 2024, article id T5.10Conference paper, Oral presentation with published abstract (Other academic)
Abstract [en]

We create partially annotated datasets from field measurements for developing models and algorithms for perception and control of forest machines using artificial intelligence, simulation, and experiments on physical testbeds.  The datasets, algorithms, and trained models for object identification, 3D perception, and motion planning and control will be made publicly available through data and code-sharing repositories.

The data is recorded using forest machines and other equipment with suitable sensors operating in the forest environment. The data include the machine and crane tip position at high resolution, and event time logs (StanForD) while the vehicle operates in high-resolution laser-scanned forest areas.  For annotation, the plan is to use both CAN-bus data and audiovisual data from operators that are willing to participate in the research. Also, by fusing visual perception with operator tree characteristics input or decision, we aim to develop a method for auto-annotation, facilitating a rapid increase in labeled training data for computer vision. In other activities, images of tree plants and bark are collected.

Research questions include, how to automate the process of creating annotated datasets and train models for identifying and positioning forestry objects, such as plants, tree species, logs, terrain obstacles, and do 3D reconstruction for motion planning and control? How large and varied datasets are required for the models to handle the variability in forests, weather, light conditions, etc.? Would additional synthetic data increase model inference accuracy?

In part we focus on forwarders traversing terrain, avoiding obstacles, and loading or unloading logs, with consideration for efficiency, safety, and environmental impact. We explore how to auto-generate and calibrate forestry machine simulators and automation scenario descriptions using the data recorded in the field. The demonstrated automation solutions serve as proofs-of-concept and references, important for developing commercial prototypes and for understanding what future research should focus on.

National Category
Computer and Information Sciences Robotics and automation
Research subject
computer and systems sciences
Identifiers
urn:nbn:se:umu:diva-227458 (URN)
Conference
IUFRO 2024 - XXVI IUFRO World Congress, Stockholm, Sweden, June 23-29, 2024
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research
Available from: 2024-06-27 Created: 2024-06-27 Last updated: 2025-02-05Bibliographically approved
Wiberg, V., Wallin, E., Fälldin, A., Semberg, T., Rossander, M., Wadbro, E. & Servin, M. (2024). Sim-to-real transfer of active suspension control using deep reinforcement learning. Robotics and Autonomous Systems, 179, Article ID 104731.
Open this publication in new window or tab >>Sim-to-real transfer of active suspension control using deep reinforcement learning
Show others...
2024 (English)In: Robotics and Autonomous Systems, ISSN 0921-8890, E-ISSN 1872-793X, Vol. 179, article id 104731Article in journal (Refereed) Published
Abstract [en]

We explore sim-to-real transfer of deep reinforcement learning controllers for a heavy vehicle with active suspensions designed for traversing rough terrain. While related research primarily focuses on lightweight robots with electric motors and fast actuation, this study uses a forestry vehicle with a complex hydraulic driveline and slow actuation. We simulate the vehicle using multibody dynamics and apply system identification to find an appropriate set of simulation parameters. We then train policies in simulation using various techniques to mitigate the sim-to-real gap, including domain randomization, action delays, and a reward penalty to encourage smooth control. In reality, the policies trained with action delays and a penalty for erratic actions perform nearly at the same level as in simulation. In experiments on level ground, the motion trajectories closely overlap when turning to either side, as well as in a route tracking scenario. When faced with a ramp that requires active use of the suspensions, the simulated and real motions are in close alignment. This shows that the actuator model together with system identification yields a sufficiently accurate model of the actuators. We observe that policies trained without the additional action penalty exhibit fast switching or bang–bang control. These present smooth motions and high performance in simulation but transfer poorly to reality. We find that policies make marginal use of the local height map for perception, showing no indications of predictive planning. However, the strong transfer capabilities entail that further development concerning perception and performance can be largely confined to simulation.

Place, publisher, year, edition, pages
Elsevier, 2024
National Category
Electrical Engineering, Electronic Engineering, Information Engineering Other Physics Topics
Research subject
Physics; computer and systems sciences
Identifiers
urn:nbn:se:umu:diva-226893 (URN)10.1016/j.robot.2024.104731 (DOI)2-s2.0-85196769514 (Scopus ID)
Projects
Mistra Digital Forest
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research, Grant DIA 2017/14 #6Wallenberg AI, Autonomous Systems and Software Program (WASP)
Available from: 2024-06-23 Created: 2024-06-23 Last updated: 2024-07-03Bibliographically approved
Fälldin, A., Lundbäck, M., Servin, M. & Wallin, E. (2024). Towards autonomous forwarding using deep learning and simulation. In: : . Paper presented at IUFRO 2024 - XXVI IUFRO World Congress, Stockholm, Sweden, June 23-29, 2024. , Article ID T5.30.
Open this publication in new window or tab >>Towards autonomous forwarding using deep learning and simulation
2024 (English)Conference paper, Oral presentation with published abstract (Other academic)
Abstract [en]

Fully autonomous forwarding is a challenge, with more imminent scenarios including operator assistance, remote-controlled machines, and semi-autonomous functions. We present several subsystems for autonomous forwarding, developed using machine learning and physics simulation,

- trafficability analysis and path planning,

- autonomous driving,

- identification of logs and high quality grasp poses, and

- crane control from snapshot camera data.

Forwarding is an energy demanding process, and repeated passages with heavy equipment can damage the soil. To avoid damage and ensure efficient use of energy, it is important with a good path planning, adapted speed, and efficient loading and unloading of logs. The collection and availability of large amounts of data is increasing in the field of forestry, opening up for autonomous solutions and efficiency improvements. This is a difficult problem though, as the forest terrain is rough, and as weather, season, obstructions, and wear present challenges in collecting and interpreting sensor-data.

Our proposed subsystems assume access to pre-scanned, high-resolution elevation maps and snapshots of log piles, captured in between crane cycles by an onboard camera. By utilizing snapshots instead of a continuous image stream in the loading task, we separate image segmentation from crane control. This removes any coupling to specific vehicle models, and greatly increases the limit on computational resources and time for the challenge of image segmentation. Log piles are normally static except at the grasp moments and given good enough grasp poses, this lack of information is not necessarily a problem.

We show how snapshot image data can be used when deploying a Reinforcement Learning agent to control the crane to grasp logs in challenging piles. Given pile RGB-D images, our grasp detection model identifies high quality grasp poses, allowing for multiple logs to be loaded in each crane cycle. Further, we show that our model is able to learn to avoid obstructions in the environment such as tree stumps or boulders. We discuss the possibility of using our model to optimize the loading task over a sequence of grasps.

Finally, we discuss how the solutions can be combined in a multi-agent forwarding system with or without a human operator in-the loop.

National Category
Computer graphics and computer vision
Identifiers
urn:nbn:se:umu:diva-227464 (URN)
Conference
IUFRO 2024 - XXVI IUFRO World Congress, Stockholm, Sweden, June 23-29, 2024
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research
Available from: 2024-06-27 Created: 2024-06-27 Last updated: 2025-02-07Bibliographically approved
Wiberg, V., Wallin, E., Nordjell, T. & Servin, M. (2022). Control of rough terrain vehicles using deep reinforcement learning. IEEE Robotics and Automation Letters, 7(1), 390-397
Open this publication in new window or tab >>Control of rough terrain vehicles using deep reinforcement learning
2022 (English)In: IEEE Robotics and Automation Letters, E-ISSN 2377-3766, Vol. 7, no 1, p. 390-397Article in journal (Refereed) Published
Abstract [en]

We explore the potential to control terrain vehicles using deep reinforcement in scenarios where human operators and traditional control methods are inadequate. This letter presents a controller that perceives, plans, and successfully controls a 16-tonne forestry vehicle with two frame articulation joints, six wheels, and their actively articulated suspensions to traverse rough terrain. The carefully shaped reward signal promotes safe, environmental, and efficient driving, which leads to the emergence of unprecedented driving skills. We test learned skills in a virtual environment, including terrains reconstructed from high-density laser scans of forest sites. The controller displays the ability to handle obstructing obstacles, slopes up to 27 degrees, and a variety of natural terrains, all with limited wheel slip, smooth, and upright traversal with intelligent use of the active suspensions. The results confirm that deep reinforcement learning has the potential to enhance control of vehicles with complex dynamics and high-dimensional observation data compared to human operators or traditional control methods, especially in rough terrain.

Place, publisher, year, edition, pages
IEEE, 2022
Keywords
Artificial Intelligence, Control and Optimization, Computer Science Applications, Computer Vision and Pattern Recognition, Mechanical Engineering, Human-Computer Interaction, Biomedical Engineering, Control and Systems Engineering
National Category
Robotics and automation Computer graphics and computer vision Other Physics Topics
Research subject
Physics; Computer Science; Automatic Control
Identifiers
urn:nbn:se:umu:diva-189485 (URN)10.1109/lra.2021.3126904 (DOI)000721999500008 ()2-s2.0-85119416792 (Scopus ID)
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research, 2017/14 #6eSSENCE - An eScience CollaborationSwedish National Infrastructure for Computing (SNIC), 2021/5-234
Available from: 2021-11-12 Created: 2021-11-12 Last updated: 2025-02-05Bibliographically approved
Wallin, E. & Servin, M. (2022). Data-driven model order reduction for granular media. Computational Particle Mechanics, 9, 15-28
Open this publication in new window or tab >>Data-driven model order reduction for granular media
2022 (English)In: Computational Particle Mechanics, ISSN 2196-4378, Vol. 9, p. 15-28Article in journal (Refereed) Published
Abstract [en]

We investigate the use of reduced-order modelling to run discrete element simulations at higher speeds. Taking a data-drivenapproach, we run many offline simulations in advance and train a model to predict the velocity field from the mass distributionand system control signals. Rapid model inference of particle velocities replaces the intense process of computing contactforces and velocity updates. In coupled DEM and multibody system simulation, the predictor model can be trained to outputthe interfacial reaction forces as well. An adaptive model order reduction technique is investigated, decomposing the mediain domains of solid, liquid, and gaseous state. The model reduction is applied to solid and liquid domains where the particlemotion is strongly correlated with the mean flow, while resolved DEM is used for gaseous domains. Using a ridge regressionpredictor, the performance is tested on simulations of a pile discharge and bulldozing. The measured accuracy is about 90%and 65%, respectively, and the speed-up range between 10 and 60. 

Place, publisher, year, edition, pages
Springer Nature, 2022
National Category
Other Physics Topics Applied Mechanics
Research subject
Physics
Identifiers
urn:nbn:se:umu:diva-169604 (URN)10.1007/s40571-020-00387-6 (DOI)000616428800001 ()2-s2.0-85101460553 (Scopus ID)
Funder
Vinnova, 2019-04832eSSENCE - An eScience Collaboration
Available from: 2020-04-08 Created: 2020-04-08 Last updated: 2023-03-24Bibliographically approved
Wallin, E., Wiberg, V., Vesterlund, F., Holmgren, J., Persson, H. & Servin, M. (2022). Learning multiobjective rough terrain traversability. Journal of terramechanics, 102, 17-26
Open this publication in new window or tab >>Learning multiobjective rough terrain traversability
Show others...
2022 (English)In: Journal of terramechanics, ISSN 0022-4898, E-ISSN 1879-1204, Vol. 102, p. 17-26Article in journal (Refereed) Published
Abstract [en]

We present a method that uses high-resolution topography data of rough terrain, and ground vehicle simulation, to predict traversability. Traversability is expressed as three independent measures: the ability to traverse the terrain at a target speed, energy consumption, and acceleration. The measures are continuous and reflect different objectives for planning that go beyond binary classification. A deep neural network is trained to predict the traversability measures from the local heightmap and target speed. To produce training data, we use an articulated vehicle with wheeled bogie suspensions and procedurally generated terrains. We evaluate the model on laser-scanned forest terrains, previously unseen by the model. The model predicts traversability with an accuracy of 90%. Predictions rely on features from the high-dimensional terrain data that surpass local roughness and slope relative to the heading. Correlations show that the three traversability measures are complementary to each other. With an inference speed 3000 times faster than the ground truth simulation and trivially parallelizable, the model is well suited for traversability analysis and optimal path planning over large areas.

Place, publisher, year, edition, pages
Elsevier, 2022
Keywords
Traversability, Rough terrain vehicle, Multibody simulation, Laser scan, Deep learning
National Category
Robotics and automation
Research subject
Physics; Computerized Image Analysis; data science; Physics
Identifiers
urn:nbn:se:umu:diva-193680 (URN)10.1016/j.jterra.2022.04.002 (DOI)000805039700001 ()2-s2.0-85129503568 (Scopus ID)
Projects
Mistra Digital Forest
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research, 2017/14 #6eSSENCE - An eScience CollaborationSwedish National Infrastructure for Computing (SNIC), 2021/5-234
Available from: 2022-04-11 Created: 2022-04-11 Last updated: 2025-02-09Bibliographically approved
Wiberg, V., Wallin, E., Wadbro, E., Rosander, M. & Servin, M. (2022). Simulation-to-reality transfer to control a forwarder with active suspensions through deep reinforcement learning. In: Woodam Chung; Christian Kanzian; Peter McNeary (Ed.), Proceedings of the joint 44th annual meeting of council on forest engineering (COFE), the 54th international symposiumon forest mechanization (FORMEC), and 2022 IUFRO all-division 3 meeting: One big family - shaping our future together. Paper presented at International Conference of Forest Engineering COFE-FORMEC-IUFRO, One Big Family: Shaping Our Future Together, Corvallis, Oregon, USA, October 4-7, 2022 (pp. 138-138). Council of forest engineering
Open this publication in new window or tab >>Simulation-to-reality transfer to control a forwarder with active suspensions through deep reinforcement learning
Show others...
2022 (English)In: Proceedings of the joint 44th annual meeting of council on forest engineering (COFE), the 54th international symposiumon forest mechanization (FORMEC), and 2022 IUFRO all-division 3 meeting: One big family - shaping our future together / [ed] Woodam Chung; Christian Kanzian; Peter McNeary, Council of forest engineering , 2022, p. 138-138Conference paper, Oral presentation with published abstract (Refereed)
Abstract [en]

Automating the loaded and unloaded driving of a forwarder has the potential to reduce operational costsup to 10% in cut-to-length logging, but remains a challening and unsolved task. The complex interaction between the vehicle and terrain requires the controller to percieve its surroundings and thestate of the vehicle to plan for traversal. Because the state space is high dimensional and the systemdynamics cannot be formulated in closed form or easily approximated, traditional control methods areinadequate. Under these conditions, where learning to act in the environement is easier than learning thesystem dynamics, model free reinforcement learning is a promising option. We use deep reinforcementlearning for control of a 16-tonne forwarder with actively articulated suspensions. To efficiently gathergeneralizable experience, the control policies were safetly trained in simulation while varying severaldomain parameters. Each policy is trained during what correponds to roughly one month of real time.In simulation, the controller shows the ability to traverse rough terrains reconstruced from high-densitylaser scans and handles slopes up to 27◦. To compare the simulated to real performance we transfer thecontrol policies to the physical vehicle. Our results provide insight on how to improve policy transfer toheavy and expensive forest machines.

Place, publisher, year, edition, pages
Council of forest engineering, 2022
National Category
Robotics and automation Computer graphics and computer vision Other Physics Topics
Research subject
Physics
Identifiers
urn:nbn:se:umu:diva-199447 (URN)979-8-9855282-1-3 (ISBN)
Conference
International Conference of Forest Engineering COFE-FORMEC-IUFRO, One Big Family: Shaping Our Future Together, Corvallis, Oregon, USA, October 4-7, 2022
Available from: 2022-09-17 Created: 2022-09-17 Last updated: 2025-02-05Bibliographically approved
Lundbäck, M., Nordfjell, T., Wiberg, V., Wallin, E. & Servin, M. (2022). Traversability analysis using high-resolution laser-scans, simulation, and deep learning. In: Woodam Chung; Christian Kanzian; Peter McNeary (Ed.), Proceedings of the joint 44th annual meeting of Council on forestengineering (COFE), the 54th International symposiumon forest mechanization (FORMEC), and 2022 IUFRO ALL-division 3 meeting: one big family – shaping our future together. Paper presented at International Conference of Forest Engineering COFE-FORMEC-IUFRO, Corvallis, Oregon, USA, October 4-7, 2022 (pp. 119-120).
Open this publication in new window or tab >>Traversability analysis using high-resolution laser-scans, simulation, and deep learning
Show others...
2022 (English)In: Proceedings of the joint 44th annual meeting of Council on forestengineering (COFE), the 54th International symposiumon forest mechanization (FORMEC), and 2022 IUFRO ALL-division 3 meeting: one big family – shaping our future together / [ed] Woodam Chung; Christian Kanzian; Peter McNeary, 2022, p. 119-120Conference paper, Oral presentation with published abstract (Other academic)
Abstract [en]

Traversability is of major importance in forestry, where heavy vehicles, weighing up to 40 tons whenfully loaded, traverse rough and sometimes soft terrain. Forest remote sensing is becoming available atresolutions where surface roughness and slope can be determined at length-scales smaller than the forestmachines. Using 3D multibody dynamics simulation of a forest machine driving in virtual terrain replications, the interaction can be captured in great detail. The observed traversability is then automaticallya function of the vehicle geometry, dynamics, and of the local terrain topography relative to heading. Weexpress traversability with three complementary measures: i) the ability to traverse the terrain at a target speed, ii) energy consumption, and iii) machine body acceleration. For high traversability, the lattertwo should be as small as possible while the first measure is at maximum. The simulations are, however,too slow for systematically probing the traversability over large areas. Instead, a deep neural networkis trained to predict the traversability measures from the local heightmap and target speed. The trainingdata comes from simulations of an articulated vehicle with wheeled bogie suspensions driving over procedurally generated terrains while observing the dynamics and local terrain topology. We evaluate themodel on laser-scanned forest terrains, previously unseen by the model. The model predicts traversability with an accuracy of 90% on terrains with 0.25 m resolution and it is 3000 times faster than the groundtruth realtime simulation and trivially parallelizable, making it well suited for traversability analysis andoptimal route planning over large areas. The trained model depends on the vehicle heading, target speed,and detailed features in the topography that a model based only on local slope and roughness cannotcapture. We explore traversability statistics over large areas of laser-scanned terrains and discuss howthe model can be used as a complement or in place of the currently used terrain classification scheme.

National Category
Computer graphics and computer vision Robotics and automation Other Physics Topics
Research subject
Physics
Identifiers
urn:nbn:se:umu:diva-199448 (URN)979-8-9855282-1-3 (ISBN)
Conference
International Conference of Forest Engineering COFE-FORMEC-IUFRO, Corvallis, Oregon, USA, October 4-7, 2022
Funder
Mistra - The Swedish Foundation for Strategic Environmental Research, 2017/14 #6
Available from: 2022-09-17 Created: 2022-09-17 Last updated: 2025-02-05Bibliographically approved
Organisations

Search in DiVA

Show all publications