Umeå University's logo

We present FORWARD, a high-resolution multimodal dataset of a cut-to-length forwarder operating in rough terrain on two harvest sites in the middle part of Sweden. The forwarder is a large Komatsu model equipped with vehicle telematics sensors, including global positioning via satellite navigation, movement sensors, accelerometers, and engine sensors. The forwarder was additionally equipped with cameras, operator vibration sensors, and multiple Inertial Measurement Units (IMUs). The data includes event time logs recorded at 5 Hz of driving speed, fuel consumption, machine position with centimeter accuracy, and crane use while the forwarder operates in forest areas, aerially laser-scanned with a resolution of around 1500 points per square meter. Production log files (Standard for Forestry Data, StanForD) with time-stamped machine events, extensive video material, and terrain data in various formats are included as well. About 18 hours of regular wood extraction work during three days is annotated from 360°-video material into individual work elements and included in the dataset. We also include scenario specifications of conducted experiments on forest roads and in terrain. Scenarios include repeatedly driving the same routes with and without steel tracks, different load weights, and different target driving speeds. The dataset is intended for developing models and algorithms for trafficability, perception, and autonomous control of forest machines using artificial intelligence, simulation, and experiments on physical testbeds. In part, we focus on forwarders traversing terrain, avoiding or handling obstacles, and loading or unloading logs, with consideration for efficiency, fuel consumption, safety, and environmental impact. Other benefits of the open dataset include the ability to explore auto-generation and calibration of forestry machine simulators and automation scenario descriptions using the data recorded in the field. The data and scripts for data exploration and analysis are made long-term publicly available through the Swedish National Data Service.

Multi-object grasping is a challenging task. It is important for energy and cost-efficient operation of industrial crane manipulators, such as those used to collect tree logs from the forest floor and on forest machines. In this work, we used synthetic data from physics simulations to explore how data-driven modeling can be used to infer multi-object grasp poses from images. We showed that convolutional neural networks can be trained specifically for synthesizing multi-object grasps. Using RGB-Depth images and instance segmentation masks as input, a U-Net model outputs grasp maps with the corresponding grapple orientation and opening width. Given an observation of a pile of logs, the model can be used to synthesize and rate the possible grasp poses and select the most suitable one, with the possibility to respect changing operational constraints such as lift capacity and reach. When tested in simulation on previously unseen data, the proposed model found successful grasp poses with an accuracy of up to 96%.

We present a simulation framework for lunar construction work involving multiple autonomous machines. The framework supports modelling of construction scenarios and autonomy solutions, execution of the scenarios in simulation, and analysis of work time and energy consumption throughout the construction project. The simulations are based on physics-based models for contacting multibody dynamics and deformable terrain, including vehicle-soil interaction forces and soil flow in real time. A behaviour tree manages the operational logic and error handling, which enables the representation of complex behaviours through a discrete set of simpler tasks in a modular hierarchical structure. High-level decision-making is separated from lower-level control algorithms, with the two connected via ROS2. Excavation movements are controlled through inverse kinematics and tracking controllers. The framework is tested and demonstrated on two different lunar construction scenarios that involve an excavator and dump truck with actively controlled articulated crawlers. 

We envision an integrated process for developing lunar construction equipment, where physical design and control are explored in parallel. In this paper, we describe a technical framework that supports this process. It relies on OpenPLX, a readable/writable declarative language that links CAD-models and autonomous systems to high-fidelity, real-time 3D simulations of contacting multibody dynamics, machine regolith interaction forces, and non-ideal sensors. To demonstrate its capabilities, we present two case studies, including an autonomous lunar rover that combines a vision-language model for navigation with a reinforcement learning-based control policy for locomotion. 

This paper presents the development and calibration of a digital twin for a wheel loader that integrates a physical machine with a high-fidelity virtual model. The digital twin supports automated diagnostics, operational optimisation, and predictive simulations to enhance construction efficiency. Calibration using experimental data from the physical wheel loader improves the model’s accuracy, ensuring realistic replication of system mechanics. A physics-based multibody dynamics model was developed in AGX Dynamics as the core digital model. The physical loader was instrumented with pressure transducers on hydraulic cylinders, load pins to measure bucket forces in excavation, a quadrature encoder for linear displacement, and an inclinometer for bucket orientation. Data from actual operations were used to calibrate the digital model so that simulated excavation forces matched experimental measurements. Results show that while the uncalibrated model accurately predicts bucket loads before excavation, significant deviations occur during soil interactions. Calibration effectively mitigates these discrepancies, yielding a validated digital twin capable of accurate excavation-force prediction and reliable performance analysis. 

Safety berms (also called safety bunds or windrows), widely employed in surface mining and quarry operations, are typically designed based on rules of thumb. Despite having been used by the industry for more than half a century and accidents happening regularly, their behaviour is still poorly understood. This paper challenges existing practices through a comprehensive investigation combining full-scale experiments and advanced numerical modelling. Focusing on a Volvo A45G articulated dump truck (ADT) and a CAT 773B rigid dump truck (RDT), collision scenarios under various approach conditions and different safety berm geometries and materials are rigorously examined. The calibrated numerical model is used to assess the energy absorption capacity of safety berms with different geometry and to predict a critical velocity for a specific scenario. Back analysis of an actual fatal accident indicated that an ADT adhering to the speed limit could not be stopped by the safety berm designed under current guidelines. The study highlights the importance of considering the entire geometry and the mass and volume of the material used to build the safety berm alongside the speed and approach conditions of the machinery. The findings of the study enable operators to set speed limits based on specific berm geometries or adapt safety berm designs to match speed constraints for specific machinery. This will reduce the risk of fatal accidents and improve haul road safety.

We investigate how well a physics-based simulator can replicate a real wheel loader performing bucket filling in a pile of soil. The comparison is made using field-test time series of the vehicle motion and actuation forces, loaded mass, and total work. The vehicle was modeled as a rigid multibody system with frictional contacts, driveline, and linear actuators. For the soil, we tested discrete-element models of different resolutions, with and without multiscale acceleration. The spatiotemporal resolution ranged between 50–400 mm and 2–500 ms, and the computational speed was between 1/10,000 to 5 times faster than real time. The simulation-to-reality gap was found to be around 10% and exhibited a weak dependence on the level of fidelity, e.g., compatible with real-time simulation. Furthermore, the sensitivity of an optimized force-feedback controller under transfer between different simulation domains was investigated. The domain bias was observed to cause a performance reduction of 5% despite the domain gap being about 15%.

We develop a method for offline parameter estimation of time-discrete multibody dynamics in maximal coordinates with regularized and frictional kinematic constraints. This setting leads to unobserved degrees of freedom, which we handle using joint state and parameter estimation. Our method finds the states and parameters as the solution to a nonlinear least squares optimization problem based on the inverse dynamics and the observation error. The solution is found using a Levenberg–Marquardt algorithm with derivatives from automatic differentiation and custom differentiation rules for the complementary conditions that appear due to dry frictional constraints. We reduce the number of method parameters to the choice of the time-step, regularization coefficients, and a parameter that controls the relative weighting of inverse dynamics and observation errors. We evaluate the method using synthetic and real measured data, focusing on performance and sensitivity to method parameters. In particular, we optimize over a 13-dimensional parameter space, including inertial, frictional, tilt, and motor parameters, using data from a real Furuta pendulum. Results show fast convergence, in the order of seconds, and good agreement for different time-series of recorded data over multiple method parameter choices. However, very stiff constraints may cause difficulties in solving the optimization problem. We conclude that our method can be very fast and has method parameters that are robust and easy to set in the tested scenarios.

The discrete element method (DEM) is a powerful tool for simulating granular soils, but its high computational demand often results in extended simulation times. While the effect of particle size has been extensively studied, the potential benefits of spatially scaling particle sizes are less explored. We systematically investigate a local particle refinement method’s impact on reducing computational effort while maintaining accuracy. We first conduct triaxial tests to verify that bulk mechanical properties are preserved under local particle refinement. Then, we perform pressure-sinkage and shear-displacement tests, comparing our method to control simulations with homogeneous particle size. We evaluate 36 different DEM beds with varying aggressiveness in particle refinement. Our results show that this approach, depending on refinement aggressiveness, can significantly reduce particle count by 2.3 to 25 times and simulation times by 3.1 to 43 times, with normalized errors ranging from 3.5% to 11.6% compared to high-resolution reference simulations. The approach maintains a high resolution at the soil surface, where interaction is high, while allowing larger particles below the surface. The results demonstrate that substantial computational savings can be achieved without significantly compromising simulation accuracy. This method can enhance the efficiency of DEM simulations in terramechanics applications.

Wheel loaders in mines and construction sites repeatedly load soil from a pile to load receivers. Automating this task presents a challenging planning problem since each loading’s performance depends on the pile state, which depends on previous loadings. We investigate an end-to-end optimization approach considering future loading outcomes and transportation costs between the pile and load receivers. To predict the evolution of the pile state and the loading performance, we use world models that leverage deep neural networks trained on numerous simulated loading cycles. A look-ahead tree search optimizes the sequence of loading actions by evaluating the performance of thousands of action candidates, which expand into subsequent action candidates under the predicted pile states recursively. Test results demonstrate that, over a horizon of 15 sequential loadings, the look-ahead tree search is 6% more efficient than a greedy strategy, which always selects the action that maximizes the current single loading performance, and 14% more efficient than using a fixed loading controller optimized for the nominal case.