Project Results

In this research, we present an end-to-end data-driven pipeline for determining the long-term stability status of objects within a given environment, specifically distinguishing between static and dynamic objects. Understanding object stability is key for mobile robots since long-term stable objects can be exploited as landmarks for long-term localisation. Our pipeline includes a labelling method that utilizes historical data from the environment to generate training data for a neural network. Rather than utilizing discrete labels, we propose the use of point-wise continuous label values, indicating the spatio-temporal stability of individual points, to train a point cloud regression network named LTS-NET. Our approach is evaluated on point cloud data from two parking lots in the NCLT dataset, and the results show that our proposed solution, outperforms direct training of a classification model for static vs dynamic object classification.

This paper presents a comprehensive review of ground agricultural robotic systems and applications with special focus on harvesting that span research and commercial products and results, as well as their enabling technologies. The majority of literature concerns the development of crop detection, field navigation via vision and their related challenges. Health monitoring, yield estimation, water status inspection, seed planting and weed removal are frequently encountered tasks. Regarding robotic harvesting, apples, strawberries, tomatoes and sweet peppers are mainly the crops considered in publications, research projects and commercial products. The reported harvesting agricultural robotic solutions, typically consist of a mobile platform, a single robotic arm/manipulator and various navigation/vision systems. This paper reviews reported development of specific functionalities and hardware, typically required by an operating agricultural robot harvester; they include (a) vision systems, (b) motion planning/navigation methodologies (for the robotic platform and/or arm), (c) Human-Robot-Interaction (HRI) strategies with 3D visualization, (d) system operation planning & grasping strategies and (e) robotic end-effector/gripper design. Clearly, automated agriculture and specifically autonomous harvesting via robotic systems is a research area that remains wide open, offering several challenges where new contributions can be made.

Kinesthetic teaching allows the direct skill transfer from the human to the robot and has been widely used to teach single arm tasks intuitively. In the bi-manual case, simultaneously moving both end-effectors is challenging due to the high physical and cognitive load imposed to the user. Thus, previous works on bi-manual task teaching resort to less intuitive methods by teaching each arm separately. This in turn requires motion synthesis and synchronization before execution. In this work, we leverage knowledge from the relative task space to facilitate a kinesthetic demonstration by guiding both end-effectors which is more human-like and intuitive
way for performing bi-manual tasks. Our method utilizes the notion of virtual fixtures and inertia minimization in the null space of the task. The controller is experimentally validated in a bi-manual task which involves the drawing of a preset line on a workpiece  utilizing two KUKA IIWA7 R800 robots. Results from ten participants were compared with a gravity compensation scheme  demonstrating improved performance

Automation of vineyards cultivation necessitates for mobile robots to retain accurate localization system. The
paper introduces a stereo vision-based Graph-Simultaneous Localization and Mapping (Graph-SLAM) pipeline custom-
tailored to the specificities of vineyard fields. Graph-SLAM is reinforced with a Loop Closure Detection (LCD) based on
semantic segmentation of the vine trees. The Mask R-CNN network is applied to segment the trunk regions of images, on
which unique visual features are extracted. These features are used to populate the bag of visual words (BoVWs) retained
on the formulated graph. A nearest neighbor search is applied to each query trunk-image to associate each unique feature
descriptor with the corresponding node in the graph using a voting procedure. We apply a probabilistic method to select the
most suitable loop closing pair and, upon an LCD appearance, the 3D points of the trunks are employed to estimate the loop
closure constraint to the graph. The traceable features on trunk segments drastically reduce the number of retained BoVWs,
which in turn significantly expedites the loop closure and graph optimization, rendering our method suitable for large scale mapping in vineyards. The pipeline has been evaluated on several data sequences gathered from real vineyards, in different seasons, when the appearance of vine trees vary significantly, and exhibited robust mapping in long distances.

We design a state-feedback controller to impose prescribed performance attributes on the output regulation error for uncertain nonlinear systems, in the presence of unknown time-varying delays appearing both to the state and control input signals, provided that an upper bound on those delays is known. The proposed controller achieves pre-specified minimum convergence rate and maximum steady-state error, and keeps bounded all signals in the closed-loop. We proved that the error is confined strictly within a delayed version of the constructed performance envelope, that depends on the difference between the actual state delay and its corresponding upper bound. Nevertheless, the maximum value of the output regulation error at steady-state remains unaltered, exactly as pre-specified by the constructed performance functions. Furthermore, the controller does not incorporate knowledge regarding the nonlinearities of the controlled system, and is of low-complexity in the sense that no hard calculations (analytic or numerical) are required to produce the control signal. Simulation results validate the theoretical findings.

 In this work, we present TS-Rep, a self-supervised method that learns representations from multi-modal varying-length time series sensor data from real robots. TS-Rep is based on a simple yet effective technique for triplet learning, where we randomly split the time series into two segments to form anchor and positive while selecting random subseries from the other time series in the mini-batch to construct negatives. We additionally use the nearest neighbour in the representation space to increase the diversity in the positives. For evaluation, we perform a clusterability analysis on representations of three heterogeneous robotics datasets. Then learned representations are applied for anomaly detection, and our method consistently performs well. A classifier trained on TS-Rep learned representations outperforms unsupervised methods and performs close to the fully-supervised methods for terrain classification. Furthermore, we show that TS-Rep is, on average, the fastest method to train among the baselines.

Work focused on deploying an autonomous robot in a vineyard at specific time intervals and to record sensor data:

- Long-term robust deployment of the robot in the wild navigating on a toplofical map.

- Building Agri-KITTI, a long-term database of robot sensor data spanning across various seasons.

- Adopting the database as benchmark for SLAM algorithms in agricultural environments.

Long-term autonomy is one of the most demanded capabilities looked into a robot. The possibility to perform the same task over and over on a long temporal horizon, offering a high standard of reproducibility and robustness, is appealing. Long-term autonomy can play a crucial role in the adoption of robotics systems for precision agriculture, for example in assisting humans in monitoring and harvesting crops in a large orchard. With this scope in mind, we report an ongoing effort in the long-term deployment of an autonomous mobile robot in a vineyard for data collection across multiple months. The main aim is to collect data from the same area at different points in time so to be able to analyse the impact of the environmental changes in the mapping and localisation tasks. In this work, we present a map-based localisation study taking 4 data sessions. We identify expected failures when the pre-built map visually differs from the environment’s current appearance and we anticipate LTS-Net, a solution pointed at extracting stable temporal features for improving long-term 4D localisation results.
 

Topological maps have proven to be an effective representation to be used for outdoor robot navigation. These typically consist of a set of nodes that represent physical locations of the environment and a set of edges representing the robot’s ability to move between these locations. They allow planning to be more efficient and to easily define different robot navigation behaviours depending on the location. In the literature, the topological maps are sometimes manually created in an 2d occupancy map previously built by a robot, but this is not very practical or scalable when it has to be done in a 50ha vineyard with hundreds of rows. Other works focus on the vine rows classification using mainly Color Vegetation indices, however this assumes there is a green canopy which is not always the case depending on the time of the year. Focusing only on the rows also leaves other non-traversable structures such as fences, buildings and poles unmapped. To overcome the aforementioned limitations, we propose a pipeline to use UAV imagery as an input to create a topological map of the vineyard where an AGV has to be deployed.
 

In this work, a control scheme for   approaching and unveiling  a partially occluded object of interest is proposed. The control scheme is based only on the classified point cloud obtained by the in-hand camera attached to the robot's end effector. It is shown that the proposed controller reaches in the vicinity of the object  progressively  unveiling the neighborhood of each visible point of the object of interest.  It can therefore potentially achieve the complete unveiling of the object. The proposed control scheme is evaluated through  simulations  and experiments with a UR5e robot with an in-hand RealSense camera on a mock-up vine setup for unveiling the stem of a grape cluster.

With the incorporation of autonomous robotic platforms in various areas (Industry, Agriculture, etc.), numerous mundane operations have been assisted by fully automated. From the dawn of humanity, in Agriculture, the high demanding working environment let the
development of techniques and machineries that could cope with each case. To further explore, new technologies (from high performance motors to optimization algorithms) have been implemented and tested in this field. Every cultivation season, there are several operations that contribute to the crop development and had to occur at least once. One of the above-mentioned operations is the weeding. In every cultivated crop, there are crops that developed which are not part of the cultivation. These crops, in most cases, have a negative impact to the crop and had to be removed. With traditional methods, weeding was taken place either by hand (smaller cultivations) or with the use of herbicides (larger cultivation). In the second case, the dosage and the time are pre-defined, and they are not taking into consideration the growth percentage and the weed allocation within the field.
In this work, a novel approach for intra-row (between the vine plants) weeding in real vineyard fields is developed and presented. All the experiments both for data aggregation and the algorithm testing were took place in a high value vineyard which produce numerous types of wine. The focus of this work was to implement an accurate real-time the weed detection and segmentation model using a deep learning algorithm in order to optimize the weed detection procedure at the intra-row of the vineyard. This approach consists of two essential sub-systems. The first one is the robotic platform that embeds all the necessary sensors (GPS, LiDAR, IMU, RGB camera) and the required computational power for the detection algorithm. The second one is the developed algorithm for weed detection. The developed algorithms were tested in many datasets from vineyards with different levels of weed development. In order to proper validate the algorithm, the unknown data ware acquired in different time periods with variations in both camera angle and wine varieties. The results show that the proposed technique gives promising results in various field conditions.

Weed management is one of the major challenges in viticulture, as long as weeds can cause significant yield losses and severe competition to the cultivations. In this direction, the development of an automated procedure for weed monitoring will provide useful data for understanding their management practices. In this work, a new image-based technique was developed in order to provide maps based on weeds’ height at the inter-row path of the vineyards. The developed algorithms were tested in many datasets from vineyards with different levels of weed development. The results show that the proposed technique gives promising results in various field conditions.

Robotic grasping in highly cluttered environments remains a challenging task due to the lack of collision free grasp
affordances. In such conditions, non-prehensile actions could help to increase such affordances. We propose a multi-fingered push-grasping policy that creates enough space for the fingers to wrap around an object to perform a stable power grasp, using a single primitive action. Our approach learns a direct mapping from visual observations to actions and is trained in a fully end-to-end manner. To achieve a more efficient learning, we decouple the action space by learning separately the robot hand pose and finger configuration. Experiments in simulation demonstrate that the proposed push-grasping policy achieves higher grasp success rate over baselines and it can generalize to unseen objects. Furthermore, although training is performed in simulation, the learned policy is robustly transferred to a real environment without a significant drop in success rate. Qualitative results, code, pre-trained models and simulation environments are available at https://robot-clutter.github.io/ppg.

 We consider the problem of controlling, with prescribed performance, uncertain Euler-Lagrange systems in the presence of aperiodic impulses affecting the system state. Between any two consecutive impulse time instants, we guarantee that the output tracking error converges to a predefined and arbitrarily small region of interest, within a prespecified fixed time. Furthermore, all signals in the closed-loop are bounded. The magnitude of the impulses and their time of appearance are unknown in advance. Yet a known minimum time interval is required to elapse, before the appearance of a new impulse. Simulation results clarify and verify the theoretical findings.

Trajectory tracking in the orientation space utilizing unit quaternions yields non linear error dynamics as opposed to Cartesian position.  In this work, we study trajectory tracking  in the orientation space utilizing the most popular quaternion error representations and angular velocity errors. By selecting error functions carefully we show exponential convergence in a region of attraction containing large initial errors.  We further show that under certain conditions frequently encountered in practice, the formulation respecting the geometric characteristics of the quaternion manifold and its tangent space yields linear tracking dynamics allowing us to guarantee a desired tracking performance by gain selection without tuning. Simulation and experimental results are provided.

In the last years, the use of robotics technology in agriculture is constantly increasing. Robotic platforms are the application of automation and robotics in agriculture field to relieve manual and heavy tasks from workers. These devices have already started to transform many aspects of agriculture and are hesitantly finding their way to the market. Therefore, robotic solutions which can provide alternative routes to weed management may provide a transformational enabling technology to mitigate against biotic and abiotic stresses on crop production, for example automatic weeding robots are the preferable substitute for chemical herbicide to remove weeds. One of the most impacting abiotic factors in agriculture are weeds, causing important yield loss in every cultivation. Integrated weed management coupled with the use of robotic platforms (UGVs), allows the effectively weed management, us a beneficial methodology for the environment. The detection of weed spots in a cultivation can be achieved by combining image acquisition by UGV and further processing by specific algorithms. These algorithms can be used to weeds control by autonomous robotic systems via mechanical procedures or herbicide spray.
The weed management is one of the major challenges in viticulture, as long as weeds can cause significant yield losses and severe competition to vines. One of the cheapest and effectiveness method remains the weed control with chemicals; however, several adverse effects and risks may arise. Different methods like tillage, thermal method, mulching and cover crops can be included in weed control strategy, depending on the environmental conditions, soil and crop. As it is known, the mechanical methods are the most cost-effective weed management methods in vineyards.
Monitoring weed in different vineyards will provide a useful database for understanding the weed management practices. In this direction, this paper presents a system for a weeding detection robot. The objective is to be enabling the weed detection robot to navigate autonomously between the inter-row spaces of crop for automatic weed control, reduce labor cost and time. In this paper, various of image processing techniques with the implementation of an RGB-D camera was examined in order to: i) detect the path between two rows of vineyard and ii) allocate the weeds based on various a priori characteristics. As a pre-processing state, the real time data from the RGB-D camera transformed into different color spaces in order to denote the noise that could occur. Subsequently, the examined algorithms and techniques tested in numerous of aggregated data from real vineyards with different levels of weed development. Finally, the developed algorithm tested by implementing it on a UGV platform with promising results.

Many visual scene understanding applications, especially in visual servoing settings, may require high quality object mask predictions for the accurate undertaking of various robotic tasks. In this work we investigate a setting where separate instance labels for all objects under view are required, but the available instance segmentation methods produce object masks inferior to a semantic segmentation algorithm. Motivated by the need to add instance label information to the higher fidelity semantic segmentation output, we propose an anisotropic label diffusion algorithm that propagates instance labels predicted by an instance segmentation algorithm inside the semantic segmentation masks. Our method leverages local topological and color information to propagate the instance labels, and is guaranteed to preserve the semantic segmentation mask. We evaluate our method on a challenging grape bunch detection dataset, and report experimental results that showcase the applicability of our method.

The problem of motion planning in obstacle cluttered environments is an important task in robotics. In the literature several methodologies exist to address the problem. In this work we consider using the feedback-based approach, where the solution comes from designing a controller capable of guaranteeing trajectory tracking with obstacle avoidance. Commonly, all respective studies consider simplified robot dynamics, which is usually insufficient in practical applications. In this work we focus on the collision avoidance problem with respect to a moving spherical object. We assume knowledge of a nominal controller that achieves tracking of a desired trajectory in the absence of obstacles, and we design an auxiliary control scheme to guarantee that the robot’s end-effector will always operate in a safe distance from the moving obstacle’s surface. The controller we develop does not take into account the actual robot dynamics, thus constituting a truly model-free approach. Experimental studies conducted on a KUKA LWR4+ robotic manipulator clarify and verify the proposed control scheme.

Spectroscopy is a widespread technique used in many scientific fields such as in the food production. The use of hyperspectral data and specifically in the visible and near infrared (VNIR) and in the short-wave infrared (SWIR) regions in grape production is of great interest. Due to its fine spectral resolution, hyperspectral analysis can contribute to both fruit monitoring and quality control at all stages of maturity with a simple and inexpensive way. This work presents an application of a contact probe spectrometer that covers the VNIR–SWIR spectrum (350–2500 nm) for the quantitative estimation of the wine grapes’ ripeness. A total of 110 samples of grape vine Syrah (Vitis vinifera Syrah) variety were collected over the 2020 harvest and pre-harvest seasons from Ktima Gerovassiliou located in Northern Greece. Their total soluble solids content (oBrix) was measured in-situ using a refractometer. Two different machine learning algorithms, namely partial least square regression (PLS) and random forest (RF) were applied along with several spectral pre-processing methods in order to predict the oBrix content from the VNIR–SWIR hyperspectral data. Additionally, the most important features of the spectrum were identified, as indicated by the most accurate models. The performance of the different models was examined in terms of the following metrics: coefficient of the determination (R2), root mean square error (RMSE) and ratio of performance to interquartile distance (RPIQ). The values of R2 = 0.90, RMSE =1.51 and RPIQ = 4.41 for PLS and 0.92, 1.34, 4.96 for RF respectively, indicate that by using a portable VNIR–SWIR spectrometer it is possible to estimate the wine grape maturity in-situ.

In this work, we present a comparative analysis of the trajectories estimated from various Simultaneous Localization and Mapping (SLAM) systems in a simulation environment for vineyards. Vineyard environment is challenging for SLAM methods, due to visual appearance changes over time, uneven terrain, and repeated visual patterns. For this reason, we created a simulation environment specifically for vineyards to help studying SLAM systems in such a challenging environment. We evaluated the following SLAM systems: LIO-SAM, StaticMapping, ORBSLAM2, and RTAB-MAP in four different scenarios. The mobile robot used in this study equipped with 2D and 3D lidars, IMU, and RGB-D camera (Kinect v2). The results show good and encouraging performance of RTAB-MAP in such an environment.

Mobile bimanual manipulation in a dynamic and uncertain environment requires the continuous and fast adjustment of the robot motion for the satisfaction of the constraints imposed by the task, the robot itself and the environment. We formulate the pick-and-place task as a sequence of mobile manipulation tasks with a combination of relative, global and local targets. Distributed distance sensors on the robot are utilized to sense the surroundings and facilitate collision avoidance with dynamic and static obstacles. We propose an approach to kinematically control the robot by solving a priority constrained optimization problem online. Experimental results on the YuMi bimanual robot mounted on the Ridgeback mobile platform validate the performance of the proposed approach.

Prehensile robotic grasping of a target object in clutter is challenging because, in such conditions, the target touches other objects, resulting to the lack of collision free grasp affordances. To address this problem, we propose a modular reinforcement learning method which uses continuous actions to totally singulate the target object from its surrounding clutter. A high level policy selects between pushing primitives, which are learned separately. Prior knowledge is effectively incorporated into learning, through action primitives and feature selection, increasing sample efficiency. Experiments demonstrate that the proposed method considerably outperforms the state-of-the-art methods in the singulation task. Furthermore, although training is performed in simulation the learned policy is robustly transferred to a real environment without a significant drop in success rate. Finally, singulation tasks in different environments are addressed by easily adding a new primitive and by retraining only the high level policy

Extracting a known target object from a pile of other objects in a cluttered environment is a challenging robotic manipulation task encountered in many robotic applications. In such conditions, the target object touches or is covered by adjacent obstacle objects, thus rendering traditional grasping techniques ineffective. In this paper, we propose a pushing policy aiming at singulating the target object from its surrounding clutter, by means of lateral pushing movements of both the neighboring objects and the target object until sufficient 'grasping room' has been achieved. To achieve the above goal we employ reinforcement learning and particularly Deep Q-learning (DQN) to learn optimal push policies by trial and error. A novel Split DQN is proposed to improve the learning rate and increase the modularity of the algorithm. Experiments show that although learning is performed in a simulated environment the transfer of learned policies to a real environment is effective thanks to robust feature selection. Finally we demonstrate that the modularity of the algorithm allows the addition of extra primitives without retraining the model from scratch.