Problems with Large Focal Lengths

[ambarak]

First of all, thanks for this cool API, it's an awesome sandbox.

The problem I would like help with, I have a collection of 47 images taken with a camera of focal length 800mm
Theia manages to get through two_view_geometric_verification, and bundle adjust the two_view_info with reasonable results, but then starts to give me the following messages:
I0425 12:49:04.235623 31351 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.

and it goes on like that for awhile before giving:

I0425 12:49:04.320199 31351 localize_view_to_reconstruction.cc:240] Failed to localize view id 0 with only 0 out of 31 features as inliers.
I0425 12:49:04.321336 31351 reconstruction_builder.cc:384]
Reconstruction estimation statistics:
Num estimated views = 2
Num input views = 47
Num estimated tracks = 295
Num input tracks = 12067
Pose estimation time = 0.003196
Triangulation time = 0
Bundle Adjustment time = 0
Total time = 0.258344

Incremental Reconstruction Estimator timings:
Time to find an initial seed for the reconstruction: 0.168333
I0425 12:49:04.340247 31351 sparse_reconstruction.cc:542]
Num views: 2
Num 3D points: 295
I0425 12:49:04.340484 31351 print_reconstruction_statistics.h:83]
Num observations: 590
Num reprojections behind camera: 0
Mean reprojection error = 0.742716
Median reprojection_error = 0.492501
I0425 12:49:04.340512 31351 print_reconstruction_statistics.h:112] Mean track length: 2
I0425 12:49:04.340518 31351 print_reconstruction_statistics.h:119] Median track length: 2
I0425 12:49:04.340534 31351 print_reconstruction_statistics.h:123] Track length histogram =
[2 - 3) = 295
[3 - 4) = 0
[4 - 5) = 0
[5 - 6) = 0
[6 - 7) = 0
[7 - 8) = 0
[8 - 9) = 0
[9 - 10) = 0
[10 - 15) = 0
[15 - 20) = 0
[20 - 25) = 0
[25 - 50) = 0

I understand the problems of dealing with narrow field of view. What I'd like help with is advise on where and how to modify Theia to try and solve this. knowing that I have very accurate gps and orientation of the cameras, plus focal length.

Thanks! in advance.

[sweeneychris]

Did you try using incremetal Sfm?

[ambarak]

Yes I have,

What looks interesting is, pair-wise Theia seems to do well in finding the correct relative pose (compared to actual GPS and IMU measurements). But it gets stumped on localization.
The images were captured in an orbital path around the scene, a similar collection taken on the same day minutes before, with a 400mm and same path, was able to reconstruct nicely. So, my assumption is, doubling the focal length to 800mm is whats causing the problem. A in-accurate initial guess for focal length could cause this, except estimate_relative_pose shows good results using given calibration data.
These are the results for all 3 sfm types, Global, Incremental and Hybrid (I can share my log files if you'd like to take a look?).

First, Global:

Reconstruction estimation statistics:
Num estimated views = 28
Num input views = 47
Num estimated tracks = 1576
Num input tracks = 17746
Pose estimation time = 0.060643
Triangulation time = 2.55008
Bundle Adjustment time = 7.275
Total time = 9.91193

Global Reconstruction Estimator timings:
Initial view graph filtering time = 6.7e-05
Camera intrinsic calibration time = 8e-06
Rotation estimation time = 0.032925
Rotation filtering time = 0.000396
Relative translation optimization time = 0.012809
Relative translation filtering time = 0.00224
Position estimation time = 0.012273
I0429 06:41:11.449508 8231 sparse_reconstruction.cc:542]
Num views: 28
Num 3D points: 1576
I0429 06:41:11.450805 8231 print_reconstruction_statistics.h:83]
Num observations: 3152
Num reprojections behind camera: 0
Mean reprojection error = 0.615282
Median reprojection_error = 0.408156
I0429 06:41:11.450884 8231 print_reconstruction_statistics.h:112] Mean track length: 2
I0429 06:41:11.450893 8231 print_reconstruction_statistics.h:119] Median track length: 2
I0429 06:41:11.450907 8231 print_reconstruction_statistics.h:123] Track length histogram =
[2 - 3) = 1576
[3 - 4) = 0
[4 - 5) = 0
[5 - 6) = 0
[6 - 7) = 0
[7 - 8) = 0
[8 - 9) = 0
[9 - 10) = 0
[10 - 15) = 0
[15 - 20) = 0
[20 - 25) = 0
[25 - 50) = 0

Second, Incremental:

I0429 06:51:36.625181 11959 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:51:36.625186 11959 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:51:36.625192 11959 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:51:36.625197 11959 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:51:36.625214 11959 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:51:36.625221 11959 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:51:36.625238 11959 localize_view_to_reconstruction.cc:240] Failed to localize view id 0 with only 0 out of 31 features as inliers.
I0429 06:51:36.627971 11959 reconstruction_builder.cc:384]
Reconstruction estimation statistics:
Num estimated views = 2
Num input views = 47
Num estimated tracks = 259
Num input tracks = 18682
Pose estimation time = 0.006479
Triangulation time = 0
Bundle Adjustment time = 0
Total time = 0.406825

Incremental Reconstruction Estimator timings:
Time to find an initial seed for the reconstruction: 0.148771
I0429 06:51:36.672530 11959 sparse_reconstruction.cc:542]
Num views: 2
Num 3D points: 259
I0429 06:51:36.672873 11959 print_reconstruction_statistics.h:83]
Num observations: 518
Num reprojections behind camera: 0
Mean reprojection error = 0.970799
Median reprojection_error = 0.745175
I0429 06:51:36.672920 11959 print_reconstruction_statistics.h:112] Mean track length: 2
I0429 06:51:36.672930 11959 print_reconstruction_statistics.h:119] Median track length: 2
I0429 06:51:36.672950 11959 print_reconstruction_statistics.h:123] Track length histogram =
[2 - 3) = 259
[3 - 4) = 0
[4 - 5) = 0
[5 - 6) = 0
[6 - 7) = 0
[7 - 8) = 0
[8 - 9) = 0
[9 - 10) = 0
[10 - 15) = 0
[15 - 20) = 0
[20 - 25) = 0
[25 - 50) = 0

Last, Hybrid:

I0429 06:57:23.620947 12252 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:57:23.620959 12252 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:57:23.620965 12252 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:57:23.620970 12252 perspective_three_point.cc:202] The 3 world points are collinear! No solution for absolute pose exits.
I0429 06:57:23.620985 12252 localize_view_to_reconstruction.cc:240] Failed to localize view id 7 with only 0 out of 30 features as inliers.
I0429 06:57:23.623701 12252 reconstruction_builder.cc:384]
Reconstruction estimation statistics:
Num estimated views = 2
Num input views = 47
Num estimated tracks = 1043
Num input tracks = 18465
Pose estimation time = 0.008786
Triangulation time = 0
Bundle Adjustment time = 0
Total time = 1.34915

Hybrid Reconstruction Estimator timings:
Time to find an initial seed for the reconstruction: 1.00586
I0429 06:57:23.685432 12252 sparse_reconstruction.cc:542]
Num views: 2
Num 3D points: 1043
I0429 06:57:23.686564 12252 print_reconstruction_statistics.h:83]
Num observations: 2086
Num reprojections behind camera: 0
Mean reprojection error = 0.539039
Median reprojection_error = 0.365783
I0429 06:57:23.686789 12252 print_reconstruction_statistics.h:112] Mean track length: 2
I0429 06:57:23.686806 12252 print_reconstruction_statistics.h:119] Median track length: 2
I0429 06:57:23.686830 12252 print_reconstruction_statistics.h:123] Track length histogram =
[2 - 3) = 1043
[3 - 4) = 0
[4 - 5) = 0
[5 - 6) = 0
[6 - 7) = 0
[7 - 8) = 0
[8 - 9) = 0
[9 - 10) = 0
[10 - 15) = 0
[15 - 20) = 0
[20 - 25) = 0
[25 - 50) = 0

And this is the flag file I used for all three, except the change the reconstruction estimator :

//############## Optimize For ###############
// SIMPLIFIED_GEOMETRIC | PINHOLE | PINHOLE_RADIAL_TANGENTIAL | FISHEYE | FOV | DIVISION_UNDISTORTION
--camera_model=PINHOLE
// FOCAL_LENGTH | ASPECT_RATIO | SKEW | PRINCIPAL_POINTS | RADIAL_DISTORTION | TANGENTIAL_DISTORTION | DISTORTION_CENTER | SCALE_FACTOR
--intrinsics_to_optimize=FOCAL_LENGTH|ASPECT_RATIO

//############## Feature Extraction ###############
--descriptor=SIFT
--feature_density=DENSE

//############## Matching Options ###############
--matching_strategy=CASCADE_HASHING
--lowes_ratio=0.75
--min_num_inliers_for_valid_match=20
--max_sampson_error_for_verified_match=10
--bundle_adjust_two_view_geometry=true
--keep_only_symmetric_matches=true

//############## Global Feature Extraction ###############
--select_image_pairs_with_global_image_descriptor_matching=true
--num_nearest_neighbors_for_global_descriptor_matching=25
--num_gmm_clusters_for_fisher_vector=16
--max_num_features_for_fisher_vector_training=1000000

//############## General SfM Options ###############
--reconstruction_estimator=HYBRID
--min_track_length=2
--max_track_length=50
--reconstruct_largest_connected_component=true
--shared_calibration=true
--only_calibrated_views=true

//############## Global SfM Options ###############
--global_position_estimator=LEAST_UNSQUARED_DEVIATION
--global_rotation_estimator=ROBUST_L1L2
--post_rotation_filtering_degrees=15.0
--refine_relative_translations_after_rotation_estimation=true
--extract_maximal_rigid_subgraph=false
--filter_relative_translations_with_1dsfm=true
--position_estimation_min_num_tracks_per_view=0
--position_estimation_robust_loss_width=0.1
--refine_camera_positions_and_points_after_position_estimation=true
--num_retriangulation_iterations=1

//############## Incremental SfM Options ###############
--absolute_pose_reprojection_error_threshold=4
--partial_bundle_adjustment_num_views=2
--full_bundle_adjustment_growth_percent=2
--min_num_absolute_pose_inliers=20

//############## Bundle Adjustment Options ###############
--bundle_adjustment_robust_loss_function=HUBER
--bundle_adjustment_robust_loss_width=15
--max_reprojection_error_pixels=5

//############## Track Subsampling Options ###############
--subsample_tracks_for_bundle_adjustment=false
--track_subset_selection_long_track_length_threshold=10
--track_selection_image_grid_cell_size_pixels=100
--min_num_optimized_tracks_per_view=100

//############## Triangulation Options ###############
--min_triangulation_angle_degrees=4
--triangulation_reprojection_error_pixels=15
--bundle_adjust_tracks=true

//############## Logging Options ###############
--alsologtostderr
--v=2