Evaluation

To evaluate the quality in an estimated camera trajectory it is collated with the ground truth and examined with different metric methods.

To evaluate the results a set of tools, provided by the Computer Vision Group TUM [1], is used. With the two scripts‚ evaluate_ate.py and evaluate_rpe.py the absolute trajectory error (ATE) and the relative pose error (RPE) are computed. In our case exemplary ground truth data is generated, where the trajectories respectively the poses (quaternions) were altered with a normal distributed noise.

Absolute Trajectory Error (ATE)

The ATE directly measures the difference between points of the true and the estimated trajectory. As a pre-processing step, the estimated poses are associated with the ground truth poses using the time stamps. Based on this association, we align the true and the estimated trajectory using singular value decomposition. Finally, the difference between each pair of poses are computed, and the mean/median/standard deviation of these differences are put out. [1]

Relative Pose Error (RPE)

This script computes the error in the relative motion between pairs of time stamps. By default, the script computes the error between all pairs of time stamps in the estimated trajectory file. As the number of time stamp pairs in the estimated trajectory is quadratic in the length of the trajectory, it can make sense to downsample this set to a fixed number (–max_pairs). Alternatively, one can choose to use a fixed window size (–fixed_delta). In this case, each pose in the estimated trajectory is associated with a later pose according to the window size (–delta) and unit (–delta_unit). This evaluation technique is useful for estimating the drift. [1]

Step-by-Step Procedure

  • make sure the format of the input files is:
    “Timestamp x-coord. y-coord. z-coord. x-quat. y-quat. z-quat. w-quat.”
    (coord. = coordinate, quat. = quaternion)
  • to execute the ATE or RPE script, the associate.pyscript is required
  • execute the ATE/RPE script with the ground-truth-file and the measurement file as arguments
  • to plot the trajectory into an image, the plot_trajectory_into_image.py script is required

Download evaluation scripts.

Examplary Trajectory with Artificial Noise

Dataset title Outdoor_static_circle
Evaluation method ATE
36705 compared pose pairs		 RPE (only rotations)
10000 compared pose pairs
Result rmse = 0.017285 m
mean = 0.015933 m
median = 0.015372 m
std = 0.006701 m
min = 0.000387 m
max = 0.050280 m		 rmse = 0.028157 deg
mean = 0.025904 deg
median = 0.025095 deg
std = 0.011035 deg
min = 0.000000 deg
max = 0.073675 deg
Ground Truth
Estimated

[1] http://vision.in.tum.de/data/datasets/rgbd-dataset/tools#evaluation [last visit 02.05.2016] (all scripts mentioned above are obtained from here)