Geotagging of Objects

Neural Networks based Solutions for Geotagging of Objects

Rozenn Dahyot

IEEE WISP webinar

2025-04-08

Introduction

timeline
    title  Some (online Data) Context
    2001: Google Earth
        : satellite & aerial
    2004: OpenStreetMap
        : GIS
        : crowd sourcing
    2005: Google Map
        : GIS
    2006: Twitter
        : Geolocated multimedia posts
    2007: Google Street View
        : Road Scenes
    2013: Mapillary  
        : Road scenes
        : crowd sourcing

Data availability, images and GIS:

• Google Earth 2001 https://en.wikipedia.org/wiki/Google_Earth
• OSM 2004 https://en.wikipedia.org/wiki/OpenStreetMap
• Google map 2005 https://en.wikipedia.org/wiki/Google_Maps
• Twitter 2006 https://en.wikipedia.org/wiki/Twitter
• GSV 2007 https://en.wikipedia.org/wiki/Google_Street_View
• Mapillary 2013 https://en.wikipedia.org/wiki/Mapillary

Introduction

timeline
    title  My research: imagery & GIS
    1998-2001: PhD 
             : Road scene video analysis
    2014-2018: GraiSearch  (FP7)
             : Social media       
    2016-2020: Bonseyes (H2020)
             : A.I. CNN
    2017: Eir 
        : Telegraph poles in ROI
        : road scene imagery (GSV)
    2018: OSI 
        : aerial imagery
        : roof tops
    2019-.: aimapit.com
        : collaborator

Introduction

Motivations:

• Infrastructure maintenance
• Infrastructure compliance (safety)
• Planning (e.g. EV public chargers deployment)
• Autonomous robotics
• Bio diversity monitoring
• etc.

Focus of this talk:

• Static Object geotagging e.g. traffic signs, poles, trees (but \(\neq\) cars or pedestrians)

Early works

Detection of Changing Objects in Camera-in-Motion Video

• Interest: static objects on the side of the road (trees, traffic signs, poles,…)

• images captured every 5 meters: frontal view from car roof (monocular camera)

• no GPS available! but milestones https://en.wikipedia.org/wiki/Milestone used as geolocation references

• feature distribution comparisons

  • \(p(m|t)\) Vs \(p(m|t-1)\)
  • \(p(m|t)\) Vs \(p(m|t+1)\)

• extraction of changing feature distributions \(p(m| t \text{ vs } t-1)\) and \(p(m| t \text{ vs } t+1)\) for backprojections.

• features \(m\) can be pixel colour (e.g. chrominance pixel pairs (2D)) and pixel shape information (Hough Transform based features)

• good detection rate, unsupervised technique!

• but sensitive to other events causing visual changes (false alarms) e.g. cars overtaking, etc.
  

Early works

\(\Rightarrow\) low dimensional feature engineering (straight edges)

\(\Rightarrow\) pixel positions \(\color{green}{(x_i,y_i)}\) used in feature computation (\(\cong\) positional encoding)

\(\Rightarrow\) P.d.f. modelled with histograms or KDE.

Shape descriptors at pixel \(i\) with position \((x_i,y_i)\) in image \(I\) with derivatives \(I_x\) and \(I_y\): \[ \left\lbrace \begin{array}{l} \|\nabla I_i\|=\sqrt{I^2_x(x_i,y_i)+I^2_y(x_i,y_i)}\\ \theta_i=\arctan\left( \frac{I_y(x_i,y_i)}{I_x(x_i,y_i)}\right)\\ \rho_i= \color{green}{x_i} \cdot \frac{I_x(x_i,y_i)}{\|\nabla I_i\|} + \color{green}{y_i} \cdot \frac{I_y(x_i,y_i)}{\|\nabla I_i\|} \end{array} \right. \] Hough Transform estimate: \[ \hat{p}(\theta,\rho)=\frac{1}{N}\sum_{i=1}^N \frac{1}{h_{\theta_i}} k_{\theta}\left(\frac{\theta-\theta_i}{h_{\theta_i}}\right)\ R_i(\theta,\rho) \]

• Note the usage of the pixel coordinates in the computation of feature \(\rho\) (\(\cong\) positional encoding)

• Hough Transform = vintage Computer vision https://en.wikipedia.org/wiki/Hough_transform

• Radon transform use to compute the kernel \(R_i\): \[ R_i(\theta,\rho)= \int\int \delta(\rho-x\cos \theta-y\sin \theta)\ k(x-x_i)\ k(y-y_i)\ dx\ dy \]

• Many man-made objects have straight edges (tree trunks sometimes have these too)

Early works

Robust Object Recognition

(Dahyot, Charbonnier, and Heitz 2000) proposed:

• using supervised learning with a (simple) AI (PCA) using a clean dataset of traffic signs augmented with rotations.

• at test time though, out-of-distribution observations that deviate from training set need to be robustly detected and recognised.

• this was done by performing robust regression against the Principal Components using a robust loss (M-estimator loss) at test time.

• a sequence of robust loss functions was used to progressively increase the strength by which outlier pixels are rejected. See the mask image provided as a feedback 1 as a by product to the robust regression: the white pixel (weight value close to 1) are inliers, while black pixel (weight value close to 0) are the detected outlier pixels.

• For a more recent overview of these robust loss functions, read also (Barron 2019, CVPR 2019)

• the slide presents two results where our “AI” is successful (when occluding pixels are in blue) and is unsuccessful (when the same occluding pixels are now black leading to confusion to another traffic sign being recognised)

Early works

Robust Object Detection

Extension to object detection:

• several robust scores (based M-estimators) computed with a sliding window
• Bayesian probability interpretation of these robust scores.
• method robust to partial occlusion and cluttered background

(Dahyot, Charbonnier, and Heitz 2004) proposed:

• Using a sliding window, the robust recognition framework is extended to detection in an image.
• the robust heat map shows one of robust similarity metrics we have proposed using M-estimators:

  • the black pixels indicates high similarity with training images

  • the white pixels indicates low similarity with training images

  • the purple pixels indicates location where no similarity is computed as the sliding window would be step outside the scene image analysed (no padding used!)

  • note that even with a cluttered background and a partial occlusion, the object is well detected!

Early works

Viz Unreal engine:

• NN score sentiment analysis of tweet text (colour lights: blue=sad; yellow=happy)
• tweet image geolocation+orientation inferred by feature matching against GSV to find tweet image GPS and orientation
  
• 3D city construction using OSM and GSV data.

(Bulbul and Dahyot 2017) uses:

• Twitter data: a Neural Network was used for sentiment analysis of text,
• the geolocation of the tweet and by analysing the image content, the 6D camera pose is estimated (Geolocation of image and direction of view )
• 3D city reconstruction using:
  • OSM building footprint and building height
  • GSV building facade image are segmented and sticked to 3D building
• Rendering in Unreal engine
  • each tweet correspond to a light coloured by its text sentiment score (blue sad/yellow happy) and directed using the direction of the photograph in the tweet.
• In the results for Rome, tweets focuses on tourist landmarks such as the Pantheon and Monument to Victor Emmanuel II.

Geotagging of Objects

In (V. Krylov, Kenny, and Dahyot 2018) we propose:

• a Markov Random Field to merge all information and to infer unique GPS tags for the objects of interest.

• note that an object can appear multiple times across several images: the output is a unique GPS for each object (no duplicate)

• we chose a modular approach: each module is trained/fine tuned independently.

• the modular approach allows to use already trained modules (e.g. depth estimation)

• We fine tune the image segmentation module to focus on our object of interest (telegraph poles)

• Alternative approach: End-to-End learnable approach (Nassar et al. 2020, GeoGraph) has that uses a GNN in combination to CNNs

Geotagging of Objects

In (V. Krylov, Kenny, and Dahyot 2018):

• the Object of interest are telegraph poles
• Data: GSV images
• Deep Neural Networks: Convolutional Neural Network
• Poles cannot be well detected from aerial/satellite images (too thin) and Street View imagery is more suited for that task.
• A pretrained CNN is used for depth estimation as multiple view feature matching approaches dont work well with thin symmetric objects like poles.
• The CNN is trained for object segmentation
• MRF for fusion of information object GPS inference
• ROI (Region of Interest): ROI (Republic of Ireland!). The whole country was processed with one GPU boosted PC by decomposing the country into small overlapping areas.

Geotagging of Objects

MRF energy to minimize is formulated as: \[ \mathcal{U}(\mathbf{z}=\lbrace z_1,\cdots,z_{N_{\mathcal{Z}}}\rbrace,\lbrace\alpha\rbrace)=\sum_{i=1}^{N_{\mathcal{Z}}} \sum_{j} \alpha_j \ \underbrace{u_j(z_i) }_{\text{energy term } j} \]

• Each image detection(s) corresponds to a ray with origin camera GPS and direction derived from pixel location and camera orientation
• Any pair of intersecting rays (from pair of images) define a site (intersection) \(i\) as potential candidate for object geolocation. \(N_{\mathcal{Z}}\) is the total number of candidate sites extracted from a collection of images.
  
• \(z_i\) at site \(i\) is a binary variable:
  • \(z_i=0\) : object is absent
  • \(z_i=1\) : object is present

In (V. Krylov, Kenny, and Dahyot 2018):

• rays = half lines with starting origin camera GPS location.
• the sites of the MRF are all the intersections between 2 rays emerging from 2 camera views.
• These MRF sites define an irregular grid.

Geotagging of Objects

Energy terms \(u_j(\cdot)\):

• One that enforces consistency with the depth estimation.

• A pairwise energy term that depends on the current state \(z\) and those of its neighbours \(\lbrace z_k \rbrace\), is introduced to enforce one detection from multiple sites clustered together.

• An energy term penalizes rays that have no positive intersections: false positives or objects discovered from a single camera position.

MRF optimization:

• Energy minimization computed with an iterative conditional modes (ICM) algorithm
  
• Initial state for ICM is set as \(z_i=0,\forall i\) (all sites are empty)

Geotagging of Objects

Preprocessing:

• Clustering: Ray intersections are only considered when in a max distance of 25 meters from the origins of the rays (camera GPS)

Postprocessing:

• Clustering: Locations of positive sites found in the same vicinity (radius=1 meter) are averaged to obtain the final unique object’s geotag.

• Postprocessing step refines the final GPS location reported in the list.

Geotagging of Objects

Evaluation:

• Object Detection

Definitions

• precision= relevant retrieved instances /all retrieved instances

• recall= relevant retrieved instances / all relevant instances

• experiments on traffic lights and telegraph poles.

• for traffic lights, there are 50 to be found, 51 instance were detected including 47 actual traffic lights (TP), 4 false positive (not traffic lights). 3 traffic lights were missed (FN).

• High recall/precision (close to 1) show good performance.

Geotagging of Objects

Evaluation:

• Absolute GPS positions against ground truth

In (V. Krylov, Kenny, and Dahyot 2018):

• robust registration between detected locations and ground truth locations

• Note: Ground truth GPS difficult to get.

• importance of MRF for improving Geolocation (depth CNN is not enough).

Geotagging of Objects

This is a video on youtube demostrating the approach (V. Krylov, Kenny, and Dahyot 2018).

Geotagging of Objects

Traffic lights geolocation using GSV + Lidar (Laefer et al. 2017)

• Detection in Lidar point cloud performed by template matching using a pole-like object template (false alarm rate: high).
• The MRF energy is modified by adding a new term to take into account Lidar candidate locations near each of the MRF sites.

• GSV images processed with fully convolutional neural networks (FCNNs)
  • sementic segmentation
  • monocular depth estimation
• the MRF irregular grid of sites are found as before with pair of rays coming from image segmentation
• LiDar: Detection of potential object locations is addressed as a template matching using a pole-like object template (free-standing object with height [2,6] meters above the ground level).
• validated for traffic lights detection.

Geotagging of Objects

Evaluation was also performed with Mapillary crowdsourced images (bottom pipeline) instead of GSV (top pipeline for reference). Additional preprocessing is needed to inferred the missing image metadata in Mapilliary.

From (V. A. Krylov and Dahyot 2019)

• Due to excessive camera position noise, the estimated geolocations are less accurate when using Mapilliary than with using GSV.

Geotagging of Objects

• Pre-processing: Enhancing quality of the GSV image metadata using Structure from Motion (Camera translation T and/or rotation R estimation).
• Postprocessing: predicted object geolocation is further refined by imposing contextual geographic information extracted from OSM.

Image metadata correction	Actual	Detected	TP	Precision\(\uparrow\)	Recall\(\uparrow\)	F-measure\(\uparrow\)	Error in meters \(\downarrow\)	Error in meters (with OSM) \(\downarrow\)
None	76	94	58	0.61	0.76	0.68	2.71	2.64
T	76	89	57	0.64	0.75	0.69	2.79	2.74
R and T	76	92	54	0.57	0.72	0.64	2.53	2.48

In (Liu et al. 2021) - we further refine the pipeline with the aim to improve data input quality from GSV (bundle adjustment) and further constraint the geolocation estimates using context information from OSM.

• note that with OSM, GPS location improves with or without bundle adjustment

• Definition F-score: The highest possible value of an F-score is 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if the precision or the recall is zero.

Geotagging of Objects

Research out of the lab!

Improving DNNs

In CNN, filter weights \(\lbrace \alpha_i\rbrace\) are learnt: \[ \text{filter for convolution}=\sum_i \alpha_i \ \mathbf{e}_i \\ \text{with} \ \lbrace \mathbf{e}_i\rbrace \equiv \text{natural basis} \]

We propose Harmonic CNN layer, where the natural basis is replaced by DCT basis,that:

• replace conventional convolutional layers to produce harmonic versions of existing CNN architectures,
• can be efficiently compressed by truncating high-frequency components,
• has been validated extensively for image classification, object detection and semantic segmentation applications.

• work by Matej ulicny PhD TCD
• focus on CNN
• started out with looking at applying CNN to JPEG images (that use DCT).

Improving DNNs

Graph matching as postprocessing of DNN segmentation results. Example for 3D segmentation of brain (IBSR dataset):

About (Chopin et al. 2023)

• work by Jeremy Chopin, PhD thesis Angers University, Postdoc MU.
  
• adding graph matching to correct error of segmentation from a CNN or transformer as a post process
• efficient when small number of samples available for training.
• IBSR dataset 18 brain 3D MRI with 32 regions annotated : only 14 considered in our evaluation
• structural information is encoded in our fully connected graph
• graph matching correct oversegmentation (but not undersegmentation), and mislabelled region
• graph matching can be solved with reinforcement learning (Chopin et al. 2022)

Summary

We presented a modular pipeline for object geotagging:

• Each module is optimised individually (not end-to-end).

• Image segmentation and depth estimation are performed with DNNs

• DNN based modules have been updated overtime (i.e. architecture changes).

• Data quality is very important (e.g. metadata) for geotagging accuracy.

• For creating a new training dataset for a new object of interest, we have taken advantages of multiple approaches (e.g. vintage Computer Vision approach, or AIs e.g. SAM)

• MRF provides a flexible formalism to take advantage of multiple sources of information.

• image examples of object of interest need to be provided for supervised learning
• sometimes it also means to gather images of objects that are not of interest but similar (e.g. electric poles Vs telegraphic poles) to improve robustness of the DNNs to false positive.

Thank you! Any Questions?

Picture left to right: V. Krylov, R. Dahyot, J. Connelly & M. Ulicny (2023)

Many thanks to all my collaborators, past and present!

References

Ahmad, W., and V. A. Krylov. 2024. “Roadside Object Geolocation from Street-Level Images with Reduced Supervision.” In 2024 32nd European Signal Processing Conference (EUSIPCO), 641–45. https://doi.org/10.23919/EUSIPCO63174.2024.10715092.

Barron, J. T. 2019. “A General and Adaptive Robust Loss Function.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4326–34. https://doi.org/10.1109/CVPR.2019.00446.

Bulbul, A., and R. Dahyot. 2017. “Social Media Based 3D Visual Popularity.” Computers & Graphics 63: 28–36. https://doi.org/10.1016/j.cag.2017.01.005.

Chopin, J., J.-B. Fasquel, H. Mouchere, R. Dahyot, and I. Bloch. 2022. “QAP Optimisation with Reinforcement Learning for Faster Graph Matching in Sequential Semantic Image Analysis.” In Pattern Recognition and Artificial Intelligence, edited by Mounîm El Yacoubi, Eric Granger, Pong Chi Yuen, Umapada Pal, and Nicole Vincent, 47–58. Paris, France: Springer International Publishing. https://doi.org/10.1007/978-3-031-09037-0_5.

Chopin, J., J.-B. Fasquel, H. Mouchère, R. Dahyot, and I. Bloch. 2023. “Model-Based Inexact Graph Matching on Top of DNNs for Semantic Scene Understanding.” Computer Vision and Image Understanding, 103744. https://doi.org/10.1016/j.cviu.2023.103744.

Dahyot, R. 2009. “Statistical Hough Transform.” IEEE Transactions on Pattern Analysis and Machine Intelligence, no. 8 (August): 1502–9. https://doi.org/10.1109/TPAMI.2008.288.

Dahyot, R., P. Charbonnier, and F. Heitz. 2000. “Robust Visual Recognition of Colour Images.” In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1:685–690 vol.1. https://doi.org/10.1109/CVPR.2000.855886.

———. 2001. “Unsupervised Statistical Detection of Changing Objects in Camera-in-Motion Video.” In Proceedings 2001 International Conference on Image Processing, 1:638–41. https://doi.org/10.1109/ICIP.2001.959126.

———. 2004. “A Bayesian Approach to Object Detection Using Probabilistic Appearance-Based Models.” Pattern Analysis and Applications 7 (3): 317–32. https://doi.org/10.1007/s10044-004-0230-5.

Krylov, V. A., and R. Dahyot. 2018. “Object Geolocation Using MRF Based Multi-Sensor Fusion.” In 2018 25th IEEE International Conference on Image Processing (ICIP), 2745–49. https://doi.org/10.1109/ICIP.2018.8451458.

———. 2019. “Object Geolocation from Crowdsourced Street Level Imagery.” In ECML PKDD 2018 Workshops, edited by Carlos Alzate, Anna Monreale, Haytham Assem, Albert Bifet, Teodora Sandra Buda, Bora Caglayan, Brett Drury, et al., 79–83. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-13453-2_7.

Krylov, V., E. Kenny, and R. Dahyot. 2018. “Automatic Discovery and Geotagging of Objects from Street View Imagery.” Remote Sensing 10 (5): 661. https://doi.org/10.3390/rs10050661.

Laefer, Debra F., Saleh Abuwarda, Anh-Vu Vo, Linh Truong-Hong, and Hamid Gharibi. 2017. “2015 Aerial Laser and Photogrammetry Datasets for Dublin, Ireland’s City Center.” New York University. Center for Urban Science; Progress. https://doi.org/10.17609/N8MQ0N.

Liu, C.-J., M. Ulicny, and R. Dahyot. 2023. Context aware object geotagging. 18087227, issued 2023. https://patents.google.com/patent/US20230206402A1/en.

Liu, C.-J., M. Ulicny, M. Manzke, and R. Dahyot. 2021. “Context Aware Object Geotagging.” In Irish Machine Vision and Image Processing (IMVIP 2021). https://doi.org/10.48550/arXiv.2108.06302.

Nassar, Ahmed Samy, Stefano D’Aronco, Sébastien Lefèvre, and Jan D. Wegner. 2020. “GeoGraph: Graph-Based Multi-View Object Detection with Geometric Cues End-to-End.” In Computer Vision – ECCV 2020, edited by Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, 488–504. Cham: Springer International Publishing.

Ulicny, M., V. A. Krylov, and R. Dahyot. 2022. “Harmonic Convolutional Networks Based on Discrete Cosine Transform.” Pattern Recognition 129: 1–12. https://doi.org/10.1016/j.patcog.2022.108707.