- Dataset size: ~125G, 1.2W colorful images in 1000 classes
- Dataset size: ~125G, 1.2W colorful images in 1000 classes
- Train: 120G, 1.2W images
- Train: 120G, 1.2W images
- Test: 5G, 50000 images
- Test: 5G, 50000 images
- Data format: RGB images.
- Data format: RGB images.
- Note: Data will be processed in src/dataset.py
- Note: Data will be processed in src/dataset.py
# [Features](#contents)
# [Features](#contents)
@ -50,12 +50,12 @@ For FP16 operators, if the input data type is FP32, the backend of MindSpore wil
# [Environment Requirements](#contents)
# [Environment Requirements](#contents)
- Hardware（Ascend/GPU/CPU）
- Hardware（Ascend/GPU/CPU）
- Prepare hardware environment with Ascend, GPU or CPU processor. If you want to try Ascend, please send the [application form](https://obs-9be7.obs.cn-east-2.myhuaweicloud.com/file/other/Ascend%20Model%20Zoo%E4%BD%93%E9%AA%8C%E8%B5%84%E6%BA%90%E7%94%B3%E8%AF%B7%E8%A1%A8.docx) to ascend@huawei.com. Once approved, you can get the resources.
- Prepare hardware environment with Ascend, GPU or CPU processor. If you want to try Ascend, please send the [application form](https://obs-9be7.obs.cn-east-2.myhuaweicloud.com/file/other/Ascend%20Model%20Zoo%E4%BD%93%E9%AA%8C%E8%B5%84%E6%BA%90%E7%94%B3%E8%AF%B7%E8%A1%A8.docx) to ascend@huawei.com. Once approved, you can get the resources.
- CPU: sh run_trian.sh CPU [DATASET_PATH] [CKPT_PATH] [FREEZE_LAYER]
- CPU: sh run_trian.sh CPU [DATASET_PATH] [CKPT_PATH] [FREEZE_LAYER] [FILTER_HEAD]
`CKPT_PATH``FREEZE_LAYER` and `FILTER_HEAD` are optional, when set `CKPT_PATH`, `FREEZE_LAYER` must be set. `FREEZE_LAYER` should be in ["none", "backbone"], and if you set `FREEZE_LAYER`="backbone", the parameter in backbone will be freezed when training and the parameter in head will not be load from checkpoint. if `FILTER_HEAD`=True, the parameter in head will not be load from checkpoint.
> RANK_TABLE_FILE is HCCL configuration file when running on Ascend.
> RANK_TABLE_FILE is HCCL configuration file when running on Ascend.
> The common restrictions on using the distributed service are as follows. For details, see the HCCL documentation.
> The common restrictions on using the distributed service are as follows. For details, see the HCCL documentation.
@ -113,14 +115,14 @@ You can start training using python or shell scripts. The usage of shell scripts
Ascend: sh run_train.sh Ascend [DEVICE_NUM][VISIABLE_DEVICES(0,1,2,3,4,5,6,7)][RANK_TABLE_FILE][DATASET_PATH][CKPT_PATH](optional)[FREEZE_LAYER](optional)[FILTER_HEAD](optional)
Ascend: sh run_train.sh Ascend [DEVICE_NUM][VISIABLE_DEVICES(0,1,2,3,4,5,6,7)][RANK_TABLE_FILE][DATASET_PATH]"
exit1
fi;
if[$2 -lt 1]&&[$2 -gt 8]
if[$2 -lt 1]&&[$2 -gt 8]
then
then
echo"error: DEVICE_NUM=$2 is not in (1-8)"
echo"error: DEVICE_NUM=$2 is not in (1-8)"
@ -59,8 +78,9 @@ run_ascend()
python train.py \
python train.py \
--platform=$1\
--platform=$1\
--dataset_path=$5\
--dataset_path=$5\
--pretrain_ckpt=$6\
--pretrain_ckpt=$PRETRAINED_CKPT\
--freeze_layer=$7\
--freeze_layer=$FREEZE_LAYER\
--filter_head=$FILTER_HEAD\
&> log$i.log &
&> log$i.log &
cd ..
cd ..
done
done
@ -68,6 +88,24 @@ run_ascend()
run_gpu()
run_gpu()
{
{
if[$#=4];then
PRETRAINED_CKPT=""
FREEZE_LAYER="none"
FILTER_HEAD="False"
elif[$#=6];then
PRETRAINED_CKPT=$5
FREEZE_LAYER=$6
FILTER_HEAD="False"
elif[$#=7];then
PRETRAINED_CKPT=$5
FREEZE_LAYER=$6
FILTER_HEAD=$7
else
echo"Usage:
GPU: sh run_train.sh GPU [DEVICE_NUM][VISIABLE_DEVICES(0,1,2,3,4,5,6,7)][DATASET_PATH][CKPT_PATH](optional)[FREEZE_LAYER](optional)[FILTER_HEAD](optional)
GPU: sh run_train.sh GPU [DEVICE_NUM][VISIABLE_DEVICES(0,1,2,3,4,5,6,7)][DATASET_PATH]"
exit1
fi;
if[$2 -lt 1]&&[$2 -gt 8]
if[$2 -lt 1]&&[$2 -gt 8]
then
then
echo"error: DEVICE_NUM=$2 is not in (1-8)"
echo"error: DEVICE_NUM=$2 is not in (1-8)"
@ -94,14 +132,32 @@ run_gpu()
python ${BASEPATH}/../train.py \
python ${BASEPATH}/../train.py \
--platform=$1\
--platform=$1\
--dataset_path=$4\
--dataset_path=$4\
--pretrain_ckpt=$5\
--pretrain_ckpt=$PRETRAINED_CKPT\
--freeze_layer=$6\
--freeze_layer=$FREEZE_LAYER\
--filter_head=$FILTER_HEAD\
&> ../train.log &# dataset train folder
&> ../train.log &# dataset train folder
}
}
run_cpu()
run_cpu()
{
{
if[$#=2];then
PRETRAINED_CKPT=""
FREEZE_LAYER="none"
FILTER_HEAD="False"
elif[$#=4];then
PRETRAINED_CKPT=$3
FREEZE_LAYER=$4
FILTER_HEAD="False"
elif[$#=5];then
PRETRAINED_CKPT=$3
FREEZE_LAYER=$4
FILTER_HEAD=$5
else
echo"Usage:
CPU: sh run_train.sh CPU [DATASET_PATH]
CPU: sh run_train.sh CPU [DATASET_PATH][CKPT_PATH](optional)[FREEZE_LAYER](optional)[FILTER_HEAD](optional)"
exit1
fi;
if[ ! -d $2]
if[ ! -d $2]
then
then
echo"error: DATASET_PATH=$2 is not a directory"
echo"error: DATASET_PATH=$2 is not a directory"
@ -120,22 +176,12 @@ run_cpu()
python ${BASEPATH}/../train.py \
python ${BASEPATH}/../train.py \
--platform=$1\
--platform=$1\
--dataset_path=$2\
--dataset_path=$2\
--pretrain_ckpt=$3\
--pretrain_ckpt=$PRETRAINED_CKPT\
--freeze_layer=$4\
--freeze_layer=$FREEZE_LAYER\
--filter_head=$FILTER_HEAD\
&> ../train.log &# dataset train folder
&> ../train.log &# dataset train folder
}
}
if[$# -gt 7]||[$# -lt 4]
then
echo"Usage:
Ascend: sh run_train.sh Ascend [DEVICE_NUM][VISIABLE_DEVICES(0,1,2,3,4,5,6,7)][RANK_TABLE_FILE][DATASET_PATH][CKPT_PATH][FREEZE_LAYER]
Ascend: sh run_train.sh Ascend [DEVICE_NUM][VISIABLE_DEVICES(0,1,2,3,4,5,6,7)][RANK_TABLE_FILE][DATASET_PATH]
GPU: sh run_train.sh GPU [DEVICE_NUM][VISIABLE_DEVICES(0,1,2,3,4,5,6,7)][DATASET_PATH][CKPT_PATH][FREEZE_LAYER]
GPU: sh run_train.sh GPU [DEVICE_NUM][VISIABLE_DEVICES(0,1,2,3,4,5,6,7)][DATASET_PATH]
CPU: sh run_train.sh CPU [DATASET_PATH][CKPT_PATH][FREEZE_LAYER]"
zhaoting
4 years ago