RPCANet++ and RPCANet

This repository is the implementation of RPCANet and RPCANet++

[2025] RPCANet++: Deep Interpretable Robust PCA for Sparse Object Segmentation.

Authors: Fengyi Wu, Yimian Dai, Tianfang Zhang, Yixuan Ding, Jian Yang, Ming-Ming Cheng, and Zhenming Peng.

[2024] RPCANet: Deep Unfolding RPCA Based Infrared Small Target Detection.

Authors: Fengyi Wu#, Tianfang Zhang#, Lei Li, Yian Huang, and Zhenming Peng.

Overview
- Overall Pipeline
- Comparison of RPCANet++ and RPCANet
Get It Started
Results and Training Weights

Overview

Robust principal component analysis (RPCA) decomposes an observation matrix into low-rank background and sparse object components. This capability has enabled its application in tasks ranging from image restoration to segmentation. However, traditional RPCA models suffer from computational burdens caused by matrix operations, reliance on finely tuned hyperparameters, and rigid priors that limit adaptability in dynamic scenarios. To solve these limitations, we propose RPCANet series (RPCANet and RPCANet++), sparse object segmentation frameworks that fuse the interpretability of RPCA with efficient deep architectures.

In this repo, we provide:

1. Instructions and Codes on training and testing for RPCANet and RPCANet++ for sparse object segmentation.

2. Interpretable Model Verification Toolkit:

lowrankness and sparsity metrics;
easy-to-use visualization tools.

Overall Pipeline

Comparison of RPCANet++ and RPCANet

RPCANet++ excced RPCANet from following aspects:

Methododology:

To solve the background transmission loss, we introduce a memory-augmented module in the BAM to enhance the feature restoration.
Besides, inspired by prior assistants in mainstream optimization methods, we design a saliency-inspired prior DCPM to accurate the object extraction and accelerate convergence process.
A more detailed deduction of Lipschitz continuity on OEM.

Experiments:

Six more datasets are added: Despite adding one more dataset on infrared small target detection, we extend the application to the sparse object segmentation domain with typical application on vessel and defect images.
We provide a more profound analysis of RPCANets with novel metrics like the measurement of low rankness and sparsity, increase the post hoc interpretability of our module.

Detail Structure of RPCANet++ and RPCANet

RPCANet++

RPCANet

Get It Started

Requirements

Python 3.8
Windows10, Ubuntu18.04 or higher
NVDIA GeForce RTX 3090
pytorch 1.8.0 or higher
More details from requirements.txt

A Quick Example

conda create -n rpcanets python=3.8
conda activate rpcanets
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

Datasets

We evaluate our methods on nine different datasets across three domains:

Infrared Small Target Detection (IRSTD): NUDT-SIRST, IRSTD-1K, SIRST, SIRST-Aug.
Vessel Segmentation (VS): DRIVE, STARE, CHASE_DB1.
Defect Detection (DD): NEU-Seg, SD-saliency-900.

Please first download these datasets and place the datasets to the folder ./datasets/.

Our project has the following structure:

├──./datasets/
│    ├── Dataset_name
│    │    ├── train
│    │    │    ├── images
│    │    │    │    ├── 000001.png
│    │    │    │    ├── ...
│    │    │    ├── masks
│    │    │    │    ├── 000001.png
│    │    │    │    ├── ...
│    │    ├── test
│    │    │    ├── images
│    │    │    │    ├── 000002.png
│    │    │    │    ├── ...
│    │    │    ├── masks
│    │    │    │    ├── 000002.png
│    │    │    │    ├── ...

Training

Run run.py to perform network training:

$ python run.py

Testing

Run t_models_istd.py t_models_vs.py t_models_dd.py to generate file of the format .mat and .png:

$ python t_models_istd.py
$ python t_models_vs.py
$ python t_models_dd.py

Note: The training and inference methods are slightly different, where we refer to Link.

The file generated will be saved to ./pngResults/ that has the following structure:

├──./pngResults/
│    ├── [dataset_name]
│    │    ├── [model_name]
│    │    │   ├── img
│    │    │   │    ├── 000000.png
│    │    │   │    ├── ...
│    │    │   ├── mat
│    │    │   │    ├── 000000.mat
│    │    │   │    ├── ...

Run cal_metric_istd.py, cal_metric_vs.py, and cal_metric_dd.pyfor direct evaluation:

$ python cal_metric_istd.py

$ python cal_metric_vs.py

$ python cal_metric_dd.py

Here, we refer to Link and Link for VS task and DD task evaluation.

Interpretable Model Verification Toolkit

The dulrs package provides tools to calculate and visualize some evaluation matrix (heatmap, low-rankness, sparsity)of our models on various scenarios from different datasets.

1. Installation

First, install the package using pip:

pip install dulrs

2. Importing the Package

Import the package in your Python script:

from dulrs import dulrs_class

3. Available Functions

The package includes the following functions:

dulrs_class(model_name, model_path, num_stages=6, use_cuda=True)
dulrs_class.heatmap(img_path, data_name,output_mat,output_png)
dulrs_class.lowrank_cal(img_path, model_name, data_name, save_dir)
dulrs_class.lowrank_draw(model_name, data_name, mat_dir, save_dir)
dulrs_class.sparsity_cal(img_path, model_name, data_name, save_dir)

Please refer to dulrs_example.py or README.md in the package for more details.

4. Visualzation Results

Heatmaps

Lowrankness Measurements

Sparsity Measurements

Results and Training Weights

Visualization Results

ISTD Task

VS Task

DD Task

Numerical Results

ISTD Task

Model	NUDT-SIRST				IRSTD-1K				SIRST				SIRST-Aug				Params	FLOPs	Weight
	IoU	F1	Pd	Fa	IoU	F1	Pd	Fa	IoU	F1	Pd	Fa	IoU	F1	Pd	Fa
RPCANet	89.31	94.35	97.14	2.87	63.21	77.45	88.31	4.39	70.37	82.61	95.41	7.42	72.54	84.08	98.21	34.14	0.680	-/0.0217	weight
RPCANet++ - S3	92.70_+3.39	96.21_+1.86	98.31	1.67	64.40_+1.19	78.34_+0.98	92.78	5.56	75.52_+5.15	86.06_+3.45	100.0	8.70	73.90_+1.36	85.01_+0.93	98.90	28.64	1.435	-/0.0262	weight
RPCANet++ - S6	94.39_+5.08	97.12_+2.77	98.41	1.34	64.93_+1.72	78.73_+1.28	89.70	4.35	74.76_+4.39	85.47_+2.86	100.0	10.77	74.89_+2.35	85.44_+1.36	98.76	28.00	2.915	-/0.0471	weight
RPCANet++ - S9	93.97_+4.66	96.89_+2.54	98.41	1.44	63.23_+0.02	77.48_+0.03	89.35	4.28	72.62_+2.25	84.14_+1.53	100.0	9.57	73.13_+0.59	84.48_+0.04	97.66	32.18	4.396	-/0.0627	weight

VS Task

Methods	DRIVE						STARE						CHASE_DB1						Params	Time (s)	Weight
	Acc	Sen	Spe	AUC	F₁	IoU	Acc	Sen	Spe	AUC	F₁	IoU	Acc	Sen	Spe	AUC	F₁	IoU	(M)	CPU/GPU
RPCANet	96.87	82.81	81.91	97.56	82.15	69.74	96.61	77.08	77.71	95.68	77.15	62.82	97.11	80.88	75.46	95.49	78.00	64.00	0.680	-/0.0217	weight
RPCANet+ - S3	96.86	83.42	81.47	95.69	82.24_+0.09	69.86_+0.12	96.80	75.86	80.19	93.41	77.83_+0.68	63.71_+0.89	97.13	81.22	75.27	94.94	78.06_+0.06	64.09_+0.09	1.435	-/0.0262	weight
RPCANet+ - S6	96.91	84.11	81.45	96.23	82.58_+0.43	70.35_+0.61	96.79	74.25	80.83	92.90	77.33_+0.18	63.04_+0.22	97.35	84.41	76.23	94.90	80.07_+2.07	66.81_+2.81	2.915	-/0.0471	weight
RPCANet+ - S9	96.85	82.95	81.71	91.87	82.12_-0.03	69.69_-0.05	96.73	75.82	79.74	88.89	77.59_+0.44	63.40_+0.58	97.03	81.56	74.11	90.68	77.59_-0.41	63.45_-0.55	4.396	-/0.0627	weight

DD Task

Methods	NEU-Seg				SD-saliency-900				Params	Time (s)	Weight
	S_m	MAE	IoU	F₁	S_m	MAE	IoU	F₁	(M)	CPU/GPU
RPCANet	84.19	0.0346	76.22	86.51	87.92	0.0243	81.28	89.68	0.680	0.0200	weight
RPCANet++ - S3	85.31	0.0295	78.99_+2.77	88.26_+1.65	87.76	0.0235	81.81_+0.53	90.00_+0.32	1.435	0.0247	weight
RPCANet++ - S6	85.75	0.0273	80.32_+4.10	89.09_+2.58	88.45	0.0215	83.30_+2.02	90.89_+1.21	2.915	0.0357	weight
RPCANet++ - S9	86.84	0.0276	80.10_+3.88	88.95_+2.44	89.28	0.0208	83.80_+2.52	91.18_+1.50	4.396	0.0499	weight

If you have any problem, please do not hesitate to contact us via [email protected].

Citation

@misc{wu2025rpcanet_pp,
      title={RPCANet++: Deep Interpretable Robust PCA for Sparse Object Segmentation}, 
      author={Fengyi Wu and Yimian Dai and Tianfang Zhang and Yixuan Ding and Jian Yang and Ming-Ming Cheng and Zhenming Peng},
      year={2025},
      eprint={2508.04190},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2508.04190}, 
}

@InProceedings{Wu_2024_WACV,
    author    = {Wu, Fengyi and Zhang, Tianfang and Li, Lei and Huang, Yian and Peng, Zhenming},
    title     = {RPCANet: Deep Unfolding RPCA Based Infrared Small Target Detection},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2024},
    pages     = {4809-4818}
}

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support