  Colorectal liver metastases (CLM) significantly impact colon cancer patients,
influencing survival based on systemic chemotherapy response. Traditional
methods like tumor grading scores (e.g., tumor regression grade - TRG) for
prognosis suffer from subjectivity, time constraints, and expertise demands.
Current machine learning approaches often focus on radiological data, yet the
relevance of histological images for survival predictions, capturing intricate
tumor microenvironment characteristics, is gaining recognition. To address
these limitations, we propose an end-to-end approach for automated prognosis
prediction using histology slides stained with H&E and HPS. We first employ a
Generative Adversarial Network (GAN) for slide normalization to reduce staining
variations and improve the overall quality of the images that are used as input
to our prediction pipeline. We propose a semi-supervised model to perform
tissue classification from sparse annotations, producing feature maps. We use
an attention-based approach that weighs the importance of different slide
regions in producing the final classification results. We exploit the extracted
features for the metastatic nodules and surrounding tissue to train a prognosis
model. In parallel, we train a vision Transformer (ViT) in a knowledge
distillation framework to replicate and enhance the performance of the
prognosis prediction. In our evaluation on a clinical dataset of 258 patients,
our approach demonstrates superior performance with c-indexes of 0.804 (0.014)
for OS and 0.733 (0.014) for TTR. Achieving 86.9% to 90.3% accuracy in
predicting TRG dichotomization and 78.5% to 82.1% accuracy for the 3-class TRG
classification task, our approach outperforms comparative methods. Our proposed
pipeline can provide automated prognosis for pathologists and oncologists, and
can greatly promote precision medicine progress in managing CLM patients.
