<jupyter_start><jupyter_text>Demonstration of evaluation tools for Task 3Evaluate and analyze results for Task 3: "Detection of graticule lines intersections".<jupyter_code># Those are the imports you may need.
from icdar21_mapseg_eval.point_detection import (
    eval_pt_detect, show_predictions, show_predictions_classified, plot_f_vs_dist_curve)
# Extra imports for the demo
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Evaluation parameters<jupyter_code># This is arbitrary, for the competition `radius_limit` is set to 50 pixels.
radius_limit = 118
# Beta for the F_beta score
beta = 0.5<jupyter_output><empty_output><jupyter_text>Sample data<jupyter_code>ground_truth = np.float32([
    [5710, 1170],  # A
    [8080, 1170],  # B
    [3330, 3530],  # C
    [5710, 3550],  # D
    [8085, 3540],  # E
    [3327, 5922],  # F
    [5715, 5940],  # G
    [8085, 5942]]) # H
predicted = np.float32([
    # A
    [5710, 1170], # exact match
    # B
    [8080 + 2*radius_limit, 1170+2*radius_limit], # Match outside acceptable area
    # C
    [3330+10, 3530+10], # multiple acceptable matches
    [3330-10, 3530-10],
    [3330+10, 3530+0],
    [3330+10, 3530+30],
    # D
    [5710+10, 3550-10], # 1 good match
    # E
    [8085+radius_limit, 3540], # far match, on the edge (will be accepted)
    # F
    # Nothing, no match
    # G and H
    [(5715+8085)/2, (5940+5942)/2] # point on the perpendicular bisector of the two points
])<jupyter_output><empty_output><jupyter_text>1. Visualize raw predictions<jupyter_code>show_predictions(ground_truth, predicted, radius_limit)<jupyter_output><empty_output><jupyter_text>2. Evaluation metrics for a single image<jupyter_code># debug=True displays matches
# score is the AUC of f_beta vs distance
# details is a dataframe containing match and score information for each predicted point
# plt_data is the curve to plot
score, details, plt_data = eval_pt_detect(ground_truth, predicted, radius_limit, 
                                          beta=beta, debug=True, show_plot=True)
# A different radius_limit gives a different result
score, details, plt_data = eval_pt_detect(ground_truth, predicted, radius_limit/3, 
                                          beta=beta, debug=False, show_plot=True)
score
# Matching details provide extra help for debugging:
details<jupyter_output><empty_output><jupyter_text>Code for error types:- `0`: correct match (true positive)- `1`: extra match withing acceptable range (error)- `2`: too far -- out of acceptable range (error)  3. Curve used to compute the AUC<jupyter_code>plot_f_vs_dist_curve(plt_data, radius_limit, beta=0.5)
# We can check we obtain the same results as the auc function from scikit-learn
from sklearn.metrics import auc
auc(plt_data["Normalized distance"], plt_data["F-beta"])
score
plt_data<jupyter_output><empty_output><jupyter_text>4. Show predictions according to error type<jupyter_code># We can show predictions with a different color according to whether they are correct or not
show_predictions_classified(ground_truth, details, radius_limit)
# And you can save the figure for further study
show_predictions_classified(ground_truth, details, radius_limit, filename="out.pdf")<jupyter_output><empty_output><jupyter_text>97. More testsWe keep here some tests to show the implementation is correct.<jupyter_code># Empty prediction: zero auc
area, df, _ = eval_pt_detect(np.array([(0,0), (1,1)]), np.empty((0,2)), 10, beta=0.5, debug=True, show_plot=True)
area, df
# Empty prediction and ground truth
area, df, _ = eval_pt_detect(np.empty((0,2)), np.empty((0,2)), 10, beta=0.5, debug=True, show_plot=True)
area, df
# Empty ground truth
area, df, df_dbg = eval_pt_detect(np.empty((0,2)), np.array([(0,0), (1,1)]), 10, beta=0.5, debug=True, show_plot=True)
area, df
# 1 perfect match over 2 targets
eval_pt_detect(np.array([(0,0), (1,1)]), np.array([(1,1)]), 10, beta=0.5, debug=True, show_plot=True)
# same
eval_pt_detect(np.array([(0,0), (1,1)]), np.array([(0,0)]), 10, beta=0.5, debug=True, show_plot=True)
# 1 perfect match and 1 extra match
eval_pt_detect(np.array([(0,0)]), np.array([(0,0), (1,1)]), 10, beta=0.5, debug=True, show_plot=True)
# 1 match, 1 miss
eval_pt_detect(np.array([(0,0), (10,10)]), np.array([(2,0)]), 4, beta=0.5, debug=True, show_plot=True)
# 1 match on edge
eval_pt_detect(np.array([(0,0)]), np.array([(10,0)]), 10, beta=0.5, debug=True, show_plot=True)
# duplicate matches
area, _, _ = eval_pt_detect(np.array([(0,0)]), np.array([(0,0), (0,0)]), 10, beta=0.5, debug=True, show_plot=True)
area<jupyter_output>Considering predicted point 00: (    0.0,    0.0)
	Matches gt point 00: (    0.0,    0.0) @ 0.0
	First match (good)
	tp:  1, fn:  0, fp:  1, |expt|:1, |pred|:2, x:0.00, y:0.56, area:0.000
Considering predicted point 01: (    0.0,    0.0)
	Matches gt point 00: (    0.0,    0.0) @ 0.0
	EXTRA MATCH (noise)
Propagating to right limit
x:1.00, y:0.56, area:0.56
Score: 0.556<jupyter_text>98. Influence of the `radius_limit` parameter<jupyter_code>area, _, _ = eval_pt_detect(ground_truth, predicted, radius_limit, beta=0.5, debug=False, show_plot=True)
area
area, _, _ = eval_pt_detect(ground_truth, predicted, 50, beta=0.5, debug=False, show_plot=True)
area
area, _, _ = eval_pt_detect(ground_truth, predicted, 1, beta=0.5, debug=False, show_plot=True)
area
area, _, _ = eval_pt_detect(ground_truth, predicted, 10, beta=0.5, debug=False, show_plot=True)
area
area, _, _ = eval_pt_detect(ground_truth, predicted, 100, beta=0.5, debug=False, show_plot=True)
area
area, _, _ = eval_pt_detect(ground_truth, predicted, 1000, beta=0.5, debug=False, show_plot=True)
area<jupyter_output><empty_output><jupyter_text>99. Fuzzy testingWe also generated some random tests for extra safety.<jupyter_code>np.random.seed(42)
targets = np.float32(np.random.randint(0,10000,(5,2)))
from scipy.spatial import distance_matrix
dmat = distance_matrix(targets, targets)
dmat[np.diag_indices_from(dmat)] = np.inf
radius_limit = np.min(dmat) / 3
radius_limit
predictions = np.float32(np.random.randint(0,10000,(100,2)))
score, details, _ = eval_pt_detect(targets, predictions, radius_limit, beta=0.5, debug=False, show_plot=True)
score
show_predictions_classified(targets, details, radius_limit)
_ = eval_pt_detect(predictions, targets, radius_limit, beta=0.5, debug=False, show_plot=True)
# Evaluation against self should give maximal score
area, _, _ = eval_pt_detect(predictions, predictions, radius_limit, beta=0.5, debug=False, show_plot=True)
area<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup