Point Transformer V3

This repo is the official project repository of the paper Point Transformer V3: Simpler, Faster, Stronger and is mainly used for releasing schedules, updating instructions, sharing experiment records (containing model weight), and handling issues. The code will be updated in Pointcept v1.5.
[ Backbone ] [PTv3] - [ arXiv ] [ Bib ] [ Code ]

Sonata: Self-supervised pre-trained PTv3

🚀 Check out our latest powerful self-supervised pre-trained PTv3, Sonata, which includes various visual demos with our pre-trained PTv3.

Highlights

Apr 05, 2024: PTv3 is selected as one of the 90 Oral presentations (3.3% accepted paper, 0.78% submissions) by CVPR'24!
Feb 28, 2024: PTv3 is accepted by CVPR'24 🎉🎉🎉.
Dec 31, 2023: We released the model code of PTv3, experiment records for scratched ScanNet and ScanNet200 are now available. More will be available soon.
Dec 19, 2023: We released our project repo for PTv3, if you have any questions related to our work, please feel free to open an issue. Subscribe to our updates by filling out the form and the subscription can be canceled by editing the form.

Citation

If you find PTv3 useful to your research, please cite our work as an acknowledgment. (੭ˊ꒳ˋ)੭✧

@inproceedings{wu2024ptv3,
    title={Point Transformer V3: Simpler, Faster, Stronger},
    author={Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang},
    booktitle={CVPR},
    year={2024}
}

@inproceedings{wu2024ppt,
    title={Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training},
    author={Wu, Xiaoyang and Tian, Zhuotao and Wen, Xin and Peng, Bohao and Liu, Xihui and Yu, Kaicheng and Zhao, Hengshuang},
    booktitle={CVPR},
    year={2024}
}

@inproceedings{wu2022ptv2,
    title={Point transformer V2: Grouped Vector Attention and Partition-based Pooling},
    author={Wu, Xiaoyang and Lao, Yixing and Jiang, Li and Liu, Xihui and Zhao, Hengshuang},
    booktitle={NeurIPS},
    year={2022}
}

@misc{pointcept2023,
    title={Pointcept: A Codebase for Point Cloud Perception Research},
    author={Pointcept Contributors},
    howpublished={\url{https://github.com/Pointcept/Pointcept}},
    year={2023}
}

Installation

Requirements

PTv3 relies on FlashAttention, while FlashAttention relies on the following requirement, make sure your local Pointcept environment satisfies the requirements:

(Recommendation)

Ubuntu: 20.04 and above
CUDA: 11.6 and above
PyTorch: 1.12.0 and above

If you can not upgrade your local environment to satisfy the above-recommended requirements, the following requirement is the minimum to run PTv3 with Pointcept, and you need to disable Flash Attention to enable PTv3:

(Minimum)

Ubuntu: 18.04 and above
CUDA: 11.3 and above
PyTorch: 1.10.0 and above

Environment

Base environment

conda create -n pointcept python=3.8 -y
conda activate pointcept
conda install ninja -y
# Choose version you want here: https://pytorch.org/get-started/previous-versions/
# We use CUDA 11.8 and PyTorch 2.1.0 for our development of PTv3
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install h5py pyyaml -c anaconda -y
conda install sharedarray tensorboard tensorboardx yapf addict einops scipy plyfile termcolor timm -c conda-forge -y
conda install pytorch-cluster pytorch-scatter pytorch-sparse -c pyg -y
pip install torch-geometric

cd libs/pointops
python setup.py install
cd ../..

# spconv (SparseUNet)
# refer https://github.com/traveller59/spconv
pip install spconv-cu118  # choose version match your local cuda version

# Open3D (visualization, optional)
pip install open3d

Flash Attention

Following README in Flash Attention repo and install Flash Attention for PTv3. This installation is optional, but we recommend enabling Flash Attention for PTv3.

Data Preparation

Please further refer Pointcept readme Data Preparation section.

Quick Start

Two running scenarios

We provide two running scenarios for PTv3, Pointcept-driven and custom-framework-driven. For the former one, you only need to clone the code of Pointcept to your local and follow the Quick Start in Pointcept to run PTv3:

git clone https://github.com/Pointcept/Pointcept.git
sh scripts/train.sh -p ${INTERPRETER_PATH} -g ${NUM_GPU} -d ${DATASET_NAME} -c ${CONFIG_NAME} -n ${EXP_NAME}

For the latter scenario, we offer a distinct instance of PTv3, disassociated from our Pointcept framework. To incorporate this code into your project, clone the project repo and copy the following file/folder to your project:

git clone https://github.com/Pointcept/PointTransformerV3.git
cp model.py ${PATH_TO_YOUR_PROJECT}
cp -r serialization ${PATH_TO_YOUR_PROJECT}

Align the input dictionary defined in our model file and the model will return the encoded feature of the given batch point cloud.

Flash Attention

The full PTv3 relies on Flash Attention, while Flash Attention relies on CUDA 11.6 and above, make sure your local Pointcept environment satisfies the requirements.

If you can not upgrade your local environment to satisfy the requirements (CUDA >= 11.6), then you can disable FlashAttention by setting the model parameter enable_flash to false and reducing the enc_patch_size and dec_patch_size to a level (e.g. 128).

FlashAttention force disables RPE and forces the accuracy reduced to fp16. If you require these features, please disable enable_flash and adjust enable_rpe, upcast_attention andupcast_softmax.

Model Zoo

1. Indoor semantic segmentation

Model	Benchmark	Additional Data	Num GPUs	Val mIoU	Config	Tensorboard	Exp Record
PTv3	ScanNet	✗	4	77.6%	link	link	link
PTv3 + PPT	ScanNet	✓	8	78.5%	link	link	link
PTv3	ScanNet200	✗	4	35.3%	link	link	link
PTv3 + PPT	ScanNet200	✓ (f.t.)	4
PTv3	S3DIS (Area5)	✗	4	73.6%	link	link	link
PTv3 + PPT	S3DIS (Area5)	✓	8	75.4%	link	link	link

*Released model weights are temporarily invalid as the model structure of PTv3 is adjusted.

Example running scripts are as follows:

# Scratched ScanNet
sh scripts/train.sh -g 4 -d scannet -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base
# PPT joint training (ScanNet + Structured3D) and evaluate in ScanNet
sh scripts/train.sh -g 8 -d scannet -c semseg-pt-v3m1-1-ppt-extreme -n semseg-pt-v3m1-1-ppt-extreme

# Scratched ScanNet200
sh scripts/train.sh -g 4 -d scannet200 -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base
# Fine-tuning from  PPT joint training (ScanNet + Structured3D) with ScanNet200
# TODO

# Scratched S3DIS, S3DIS rely on RPE, also an example for disable flash attention
sh scripts/train.sh -g 4 -d s3dis -c semseg-pt-v3m1-0-rpe -n semseg-pt-v3m1-0-rpe
# PPT joint training (ScanNet + S3DIS + Structured3D) and evaluate in ScanNet
sh scripts/train.sh -g 8 -d s3dis -c semseg-pt-v3m1-1-ppt-extreme -n semseg-pt-v3m1-1-ppt-extreme

# More configs and exp records for PTv3 will be available soon.

2.Outdoor semantic segmentation

Model	Benchmark	Additional Data	Num GPUs	Val mIoU	Config	Tensorboard	Exp Record
PTv3	nuScenes	✗	4	80.3	link	link	link
PTv3 + PPT	nuScenes	✓	8
PTv3	SemanticKITTI	✗	4
PTv3 + PPT	SemanticKITTI	✓	8
PTv3	Waymo	✗	4	71.2	link	link	link (log only)
PTv3 + PPT	Waymo	✓	8

*Released model weights are temporarily invalid as the model structure of PTv3 is adjusted.
*Model weights trained with Waymo Open Dataset cannot be released due to the regulations.

Example running scripts are as follows:

# Scratched ScanNet
sh scripts/train.sh -g 4 -d scannet -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base
# PPT joint training (ScanNet + Structured3D) and evaluate in ScanNet
sh scripts/train.sh -g 8 -d scannet -c semseg-pt-v3m1-1-ppt-extreme -n semseg-pt-v3m1-1-ppt-extreme

# Scratched ScanNet200
sh scripts/train.sh -g 4 -d scannet200 -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base
# Fine-tuning from  PPT joint training (ScanNet + Structured3D) with ScanNet200
# TODO

# Scratched S3DIS, S3DIS rely on RPE, also an example for disable flash attention
sh scripts/train.sh -g 4 -d s3dis -c semseg-pt-v3m1-0-rpe -n semseg-pt-v3m1-0-rpe
# PPT joint training (ScanNet + S3DIS + Structured3D) and evaluate in ScanNet
sh scripts/train.sh -g 8 -d s3dis -c semseg-pt-v3m1-1-ppt-extreme -n semseg-pt-v3m1-1-ppt-extreme
# S3DIS 6-fold cross validation
# 1. The default configs are evaluated on Area_5, modify the "data.train.split", "data.val.split", and "data.test.split" to make the config evaluated on Area_1 ~ Area_6 respectively.
# 2. Train and evaluate the model on each split of areas and gather result files located in "exp/s3dis/EXP_NAME/result/Area_x.pth" in one single folder, noted as RECORD_FOLDER.
# 3. Run the following script to get S3DIS 6-fold cross validation performance:
export PYTHONPATH=./
python tools/test_s3dis_6fold.py --record_root ${RECORD_FOLDER}

# Scratched nuScenes
sh scripts/train.sh -g 4 -d nuscenes -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base
# Scratched Waymo
sh scripts/train.sh -g 4 -d waymo -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base

# More configs and exp records for PTv3 will be available soon.

License

Pointcept/PointTransformerV3

ErrorLooks like something went wrong!

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Error
Looks like something went wrong!

Packages