利用caffe生成 lmdb 格式的文件，并对网络进行FineTuning
数据的组织格式为：
首先，所需要的脚本指令路径为：
/home/wangxiao/Downloads/caffe-master/examples/imagenet/
其中，生成lmdb的文件为： create_imagenet.sh
接下来的主要任务就是修改自己的data的存放路径了。
1 #!/usr/bin/env sh 2 # Create the imagenet lmdb inputs 3 # N.B. set the path to the imagenet train + val data dirs 4 5 EXAMPLE=../wangxiao 6 DATA=../fine_tuning_data 7 TOOLS=../build/tools 8 9 TRAIN_DATA_ROOT=../fine_tuning_data/training/data/10 VAL_DATA_ROOT=../fine_tuning_data/validation/data/11 12 #TRAIN_DATA_ROOT=/media/yukai/247317a3-e6b5-45d4-81d1-956930526746/---------------/Attribute reconginition/final_PETA_dataset/whole_benchmark/用于微调网络的数据/training/data/13 #VAL_DATA_ROOT=/media/yukai/247317a3-e6b5-45d4-81d1-956930526746/---------------/Attribute reconginition/final_PETA_dataset/whole_benchmark/用于微调网络的数据/validation/data/14 15 # Set RESIZE=true to resize the images to 256x256. Leave as false if images have16 # already been resized using another tool.17 18 # RESIZE=false default parameter and wangxiao modify it in 2015.10.13 1:2519 20 RESIZE=true21 if $RESIZE; then22 RESIZE_HEIGHT=25623 RESIZE_WIDTH=25624 else25 RESIZE_HEIGHT=026 RESIZE_WIDTH=027 fi28 29 if [ ! -d "$TRAIN_DATA_ROOT" ]; then30 echo "Error: TRAIN_DATA_ROOT is not a path to a directory: $TRAIN_DATA_ROOT"31 echo "Set the TRAIN_DATA_ROOT variable in create_imagenet.sh to the path" \32 "where the ImageNet training data is stored."33 exit 134 fi35 36 if [ ! -d "$VAL_DATA_ROOT" ]; then37 echo "Error: VAL_DATA_ROOT is not a path to a directory: $VAL_DATA_ROOT"38 echo "Set the VAL_DATA_ROOT variable in create_imagenet.sh to the path" \39 "where the ImageNet validation data is stored."40 exit 141 fi42 43 echo "Creating train lmdb..."44 45 GLOG_logtostderr=1 $TOOLS/convert_imageset \46 --resize_height=$RESIZE_HEIGHT \47 --resize_width=$RESIZE_WIDTH \48 --shuffle \49 $TRAIN_DATA_ROOT \50 $DATA/training/final_train_data.txt \51 $EXAMPLE/PETA_train_lmdb52 53 #echo "Creating val lmdb..."54 55 #GLOG_logtostderr=1 $TOOLS/convert_imageset \56 # --resize_height=$RESIZE_HEIGHT \57 # --resize_width=$RESIZE_WIDTH \58 # --shuffle \59 # $VAL_DATA_ROOT \60 # $DATA/validation/final_test_data.txt \61 # $EXAMPLE/PETA_val_lmdb62 63 echo "Done."
都修改完成后，在终端执行：create_imagenet.sh，然后会有如此的提示，表示正在生成lmdb文件：
然后完成后，提示： Done. 然后可以看到已经生成了所需要的文件：
然后利用 make_imagenet_mean.sh 生成所需要的 mean file :
caffe-master$: sh ./make_imagenet_mean.sh
1 #!/usr/bin/env sh 2 # Compute the mean image from the imagenet training lmdb 3 # N.B. this is available in data/ilsvrc12 4 5 EXAMPLE=../wangxiao 6 DATA=./data 7 TOOLS=../build/tools 8 9 #echo $TOOLS/compute_image_mean $EXAMPLE/HAT_train_lmdb \10 # $DATA/HAT_mean.binaryproto11 $TOOLS/compute_image_mean $EXAMPLE/HAT_train_lmdb \12 $DATA/HAT_mean.binaryproto13 echo "Done."
然后就生成了 HAT_mean.binaryproto
其中，caffe路径下：/home/wangxiao/Downloads/caffe-master/examples/imagenet/readme.md 对这个过程有一个详细的解释。
然后就是查看 solver.prototxt:
1 net: "models/bvlc_reference_caffenet/train_val.prototxt" 2 test_iter: 1000 3 test_interval: 1000 4 base_lr: 0.01 5 lr_policy: "step" 6 gamma: 0.1 7 stepsize: 100000 8 display: 20 9 max_iter: 45000010 momentum: 0.911 weight_decay: 0.000512 snapshot: 1000013 snapshot_prefix: "models/bvlc_reference_caffenet/caffenet_train"14 solver_mode: GPU
打开 models/bvlc_reference_caffenet/train_val.prototxt
需要修改的也就到第55行：
1 name: "CaffeNet" 2 layer { 3 name: "data" 4 type: "Data" 5 top: "data" 6 top: "label" 7 include { 8 phase: TRAIN 9 }10 transform_param {11 mirror: true12 crop_size: 22713 mean_file: "wangxiao/HAT_data/HAT_mean.binaryproto"14 }15 # mean pixel / channel-wise mean instead of mean image16 # transform_param {17 # crop_size: 22718 # mean_value: 10419 # mean_value: 11720 # mean_value: 12321 # mirror: true22 # }23 data_param {24 source: "wangxiao/HAT_data/HAT_train_lmdb"25 batch_size: 25626 backend: LMDB27 }28 }29 layer {30 name: "data"31 type: "Data"32 top: "data"33 top: "label"34 include {35 phase: TEST36 }37 transform_param {38 mirror: false39 crop_size: 22740 mean_file: "wangxiao/HAT_data/HAT_mean.binaryproto"41 }42 # mean pixel / channel-wise mean instead of mean image43 # transform_param {44 # crop_size: 22745 # mean_value: 10446 # mean_value: 11747 # mean_value: 12348 # mirror: true49 # }50 data_param {51 source: "wangxiao/HAT_data/HAT_val_lmdb"52 batch_size: 5053 backend: LMDB54 }55 }
然后执行：
终端会有显示：
1 I0103 13:44:21.027832 9543 net.cpp:297] Network initialization done. 2 I0103 13:44:21.027839 9543 net.cpp:298] Memory required for data: 1757220868 3 I0103 13:44:21.027928 9543 solver.cpp:66] Solver scaffolding done. 4 I0103 13:44:21.028312 9543 caffe.cpp:212] Starting Optimization 5 I0103 13:44:21.028326 9543 solver.cpp:294] Solving CaffeNet 6 I0103 13:44:21.028333 9543 solver.cpp:295] Learning Rate Policy: step 7 I0103 13:44:22.012593 9543 solver.cpp:243] Iteration 0, loss = 7.52783 8 I0103 13:44:22.012660 9543 solver.cpp:259] Train net output #0: loss = 7.52783 (* 1 = 7.52783 loss) 9 I0103 13:44:22.012687 9543 solver.cpp:590] Iteration 0, lr = 0.0110 I0103 13:44:41.812361 9543 solver.cpp:243] Iteration 20, loss = 3.972311 I0103 13:44:41.812413 9543 solver.cpp:259] Train net output #0: loss = 3.9723 (* 1 = 3.9723 loss)12 I0103 13:44:41.812428 9543 solver.cpp:590] Iteration 20, lr = 0.0113 I0103 13:45:01.553021 9543 solver.cpp:243] Iteration 40, loss = 2.971514 I0103 13:45:01.553104 9543 solver.cpp:259] Train net output #0: loss = 2.9715 (* 1 = 2.9715 loss)15 I0103 13:45:01.553119 9543 solver.cpp:590] Iteration 40, lr = 0.0116 I0103 13:45:21.574745 9543 solver.cpp:243] Iteration 60, loss = 2.9154717 I0103 13:45:21.574798 9543 solver.cpp:259] Train net output #0: loss = 2.91547 (* 1 = 2.91547 loss)18 I0103 13:45:21.574811 9543 solver.cpp:590] Iteration 60, lr = 0.0119 I0103 13:45:41.247493 9543 solver.cpp:243] Iteration 80, loss = 2.9645120 I0103 13:45:41.247627 9543 solver.cpp:259] Train net output #0: loss = 2.96451 (* 1 = 2.96451 loss)21 I0103 13:45:41.247642 9543 solver.cpp:590] Iteration 80, lr = 0.0122 I0103 13:46:00.941267 9543 solver.cpp:243] Iteration 100, loss = 2.8588723 I0103 13:46:00.941318 9543 solver.cpp:259] Train net output #0: loss = 2.85887 (* 1 = 2.85887 loss)24 I0103 13:46:00.941332 9543 solver.cpp:590] Iteration 100, lr = 0.0125 I0103 13:46:20.628329 9543 solver.cpp:243] Iteration 120, loss = 2.9131826 I0103 13:46:20.628463 9543 solver.cpp:259] Train net output #0: loss = 2.91318 (* 1 = 2.91318 loss)27 I0103 13:46:20.628476 9543 solver.cpp:590] Iteration 120, lr = 0.0128 I0103 13:46:40.621937 9543 solver.cpp:243] Iteration 140, loss = 3.0649929 I0103 13:46:40.621989 9543 solver.cpp:259] Train net output #0: loss = 3.06499 (* 1 = 3.06499 loss)30 I0103 13:46:40.622004 9543 solver.cpp:590] Iteration 140, lr = 0.0131 I0103 13:47:00.557921 9543 solver.cpp:243] Iteration 160, loss = 2.981832 I0103 13:47:00.558048 9543 solver.cpp:259] Train net output #0: loss = 2.9818 (* 1 = 2.9818 loss)33 I0103 13:47:00.558063 9543 solver.cpp:590] Iteration 160, lr = 0.01
因为设置的迭代次数为： 450000次，所以，接下来就是睡觉了。。。O(∩_∩)O~ 感谢木得兄刚刚的帮助。
------ 未完待续------
另外就是，当loss 后期变化不大的时候，可以试着调整学习率， 在Solver.prototext中：
1 train_net: "models/bvlc_reference_caffenet/train_val.prototxt" 2 # test_iter: 1000 3 # test_interval: 1000 4 base_lr: 0.0001 5 lr_policy: "step" 6 gamma: 0.1 7 stepsize: 100000 8 display: 20 9 max_iter: 45000010 momentum: 0.911 weight_decay: 0.000512 snapshot: 1000013 snapshot_prefix: "models/bvlc_reference_caffenet/caffenet_train"14 solver_mode: GPU
base_lr: 0.0001 每次可以改为0.1×base_lr， 这里的 0.0001 是我两次调整之后的数值。 然后运行 resume_training.sh
1 #!/usr/bin/env sh2 3 ./build/tools/caffe train \4 --solver=models/bvlc_reference_caffenet/solver.prototxt \5 --snapshot=models/bvlc_reference_caffenet/caffenet_train_iter_88251.solverstate
将snapshot改为之前中断时的结果即可，即： caffenet_train_iter_88251.solverstate
继续看loss是否降低。。。
--------------------------------- 未完待续 ---------------------------------