Skip to content

Event-enhanced Line Segment Detection in Motion-blurred Images

License

Notifications You must be signed in to change notification settings

lh9171338/FE-LSD

Repository files navigation

FE-LSD

This repository contains the official PyTorch implementation of the paper: Detecting Line Segments in Motion-blurred Images with Events.

Introduction

FE-LSD is an event-enhanced line segment detection framework for motion-blurred images with thoughtful information fusion of both modalities and advanced wireframe parsing network. Extensive results on both synthetic and realistic datasets demonstrate the effectiveness of the proposed method for handling motion blurs in line segment detection.

Network Architecture

Results

FE-Wireframe Dataset

  • Quantitative Comparisons
Method sAP5 sAP10 sAP15 msAP mAPJ APH FH FPS
LSD 0.1 0.6 1.1 0.6 3.0 19.5 42.6 76.7
FBSD 0.2 0.4 0.9 0.5 2.9 24.9 47.0 21.7
L-CNN 3.4 5.1 6.2 4.9 7.0 22.7 38.8 28.8
HAWP 3.5 5.1 6.3 5.0 6.8 21.7 40.2 36.6
ULSD 3.5 5.3 6.8 5.2 7.5 20.2 40.3 39.7
LETR 2.8 5.0 6.5 4.8 7.3 21.9 41.9 4.2
L-CNN (Retrained) 40.6 45.8 48.2 44.8 45.6 70.5 71.1 10.6
HAWP (Retrained) 45.1 50.4 52.9 49.5 46.8 75.0 73.2 26.8
ULSD (Retrained) 47.0 52.7 55.2 51.7 48.8 72.2 73.7 32.2
LETR (Retrained) 24.7 34.7 39.7 33.1 25.4 66.1 71.5 3.9
FE-HAWP 48.7 53.9 56.2 53.0 49.4 77.1 75.1 22.2
FE-ULSD 50.9 56.5 58.8 55.4 51.1 75.3 75.9 24.2
  • Qualitative Comparisons

FE-Blurframe Dataset

  • Quantitative Comparisons
Method sAP5 sAP10 sAP15 msAP mAPJ APH FH FPS
LSD 1.1 2.8 4.1 2.7 5.1 29.4 48.1 61.0
FBSD 0.9 1.9 2.7 1.8 5.1 34.2 53.2 15.9
L-CNN 7.5 11.5 13.7 10.9 12.4 27.9 45.2 29.7
HAWP 8.4 12.8 15.3 12.2 12.4 32.0 48.2 38.1
ULSD 6.8 10.8 13.0 10.2 11.8 26.7 45.6 40.6
LETR 7.1 13.0 16.8 12.3 12.1 30.2 51.1 3.6
L-CNN (Retrained) 34.0 40.3 43.0 39.1 40.3 66.0 67.1 17.7
HAWP (Retrained) 37.0 43.9 46.9 42.6 41.6 67.9 69.6 29.0
ULSD (Retrained) 42.0 47.8 50.4 46.7 48.5 67.0 69.3 32.2
LETR (Retrained) 22.6 33.8 38.8 31.7 23.2 57.7 65.4 3.3
FE-HAWP 47.5 53.0 55.4 52.0 50.9 74.0 73.9 19.3
FE-ULSD 47.3 52.9 55.2 51.8 52.2 72.9 73.7 19.7
FE-HAWP (Fine-tuned) 59.8 64.2 65.9 63.3 60.1 82.0 79.7 21.1
FE-ULSD (Fine-tuned) 59.3 63.8 65.7 62.9 61.0 77.8 77.1 21.6
  • Qualitative Comparisons

Requirements

  • torch>=1.6.0
  • torchvision>=0.7.0
  • CUDA>=10.1
  • lh_tool, matplotlib, numpy, opencv_python, Pillow, scikit_learn, scipy, setuptools, tensorboardX, timm, torch, torchvision, tqdm, yacs,

Step-by-step installation

conda create --name FE-LSD python=3.8
conda activate FE-LSD

cd <FE-LSD-Path>
git clone https://github.com/lh9171338/FE-LSD.git
cd FE-LSD

pip install -r requirements.txt

python setup.py build_ext --inplace

Quickstart with the pretrained model

  • There are pretrained models in Google drive and Baiduyun. Please download them and put in the model/ folder.
  • Put your test data in the dataset/ folder and generate the test.json file.
python image2json.py --dataset_name <DATASET_NAME>
  • The file structure is as follows:
|-- dataset
    |-- events
	|-- 000001.npz
	|-- ...
    |-- images-blur
	|-- 000001.png
	|-- ...
    |-- test.json
  • Test with the pretrained model. The results are saved in the output/ folder.
python test.py --arch <ARCH> --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --save_image

Training & Testing

Data Preparation

  • Download the dataset from Baiduyun.
  • Unzip the dataset to the dataset/ folder.
  • Convert event streams into synchronous frames using Event Spike Tensor (EST) representation.
python event2frame.py --dataset_name <DATASET_NAME> --representation EST

# for FE-Wireframe dataset
python event2frame.py --dataset_name FE-Wireframe --representation EST

# for FE-Blurframe dataset
python event2frame.py --dataset_name FE-Blurframe --representation EST -s 0.5

ln -s events-EST-10 events

Train

python train.py --arch FE-HAWP --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> [--gpu <GPU_ID>] # FE-HAWP
python train.py --arch FE-ULSD --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> [--gpu <GPU_ID>] # FE-ULSD

Test

python test.py --arch FE-HAWP --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --save_image --with_clear [--gpu <GPU_ID>] # FE-HAWP
python test.py --arch FE-ULSD --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --save_image --with_clear [--gpu <GPU_ID>] # FE-ULSD

Evaluation

To evaluate the mAPJ, sAP, and FPS

python test.py --arch FE-HAWP --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --evaluate [--gpu <GPU_ID>] # FE-HAWP
python test.py --arch FE-ULSD --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --evaluate [--gpu <GPU_ID>] # FE-ULSD

To evaluate APH and FH, MATLAB is required

cd metric
python eval_APH.py --arch FE-HAWP --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> # FE-HAWP
python eval_APH.py --arch FE-ULSD --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> # FE-ULSD

Citation

@ARTICLE{10323537,
  author={Yu, Huai and Li, Hao and Yang, Wen and Yu, Lei and Xia, Gui-Song},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Detecting Line Segments in Motion-Blurred Images With Events}, 
  year={2023},
  pages={1-16},
  doi={10.1109/TPAMI.2023.3334877}
}

About

Event-enhanced Line Segment Detection in Motion-blurred Images

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published