-
Notifications
You must be signed in to change notification settings - Fork 45.5k
Description
I train the deeplab and use pre-trained model named 'xception_65',however,it remains me:
WARNING:tensorflow:Variable aspp2_depthwise/depthwise_weights/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/feature_projection0/weights missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv1_depthwise/BatchNorm/beta missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable image_pooling/BatchNorm/gamma missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp1_depthwise/depthwise_weights missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp0/BatchNorm/gamma missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp2_pointwise/BatchNorm/gamma missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable image_pooling/BatchNorm/beta/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp0/BatchNorm/gamma/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv1_pointwise/BatchNorm/beta missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp2_pointwise/BatchNorm/beta missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable logits/semantic/weights missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp2_depthwise/BatchNorm/gamma missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv1_pointwise/weights/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv0_pointwise/BatchNorm/beta/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv1_depthwise/BatchNorm/moving_mean missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv0_depthwise/BatchNorm/beta/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp2_pointwise/BatchNorm/gamma/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp2_depthwise/BatchNorm/moving_mean missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/feature_projection0/BatchNorm/moving_mean missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable logits/semantic/biases missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp2_pointwise/weights/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp0/BatchNorm/beta/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp1_pointwise/BatchNorm/gamma/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp1_pointwise/BatchNorm/beta/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp1_pointwise/BatchNorm/gamma missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp1_depthwise/BatchNorm/gamma/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/feature_projection0/BatchNorm/gamma/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp2_pointwise/BatchNorm/beta/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv1_depthwise/BatchNorm/gamma missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp3_pointwise/BatchNorm/gamma/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv0_depthwise/BatchNorm/moving_mean missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp3_depthwise/BatchNorm/gamma/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv0_depthwise/BatchNorm/beta missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp3_depthwise/BatchNorm/beta/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable concat_projection/BatchNorm/gamma missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable image_pooling/BatchNorm/moving_variance missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp1_depthwise/BatchNorm/beta/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp3_pointwise/BatchNorm/beta missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp2_depthwise/BatchNorm/beta missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv0_pointwise/BatchNorm/moving_mean missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv0_pointwise/BatchNorm/gamma/Momentum missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable decoder/decoder_conv1_pointwise/BatchNorm/moving_mean missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp0/weights missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt WARNING:tensorflow:Variable aspp1_pointwise/BatchNorm/beta missing in checkpoint /home/vrview/models-master/research/deeplab/xception/model.ckpt
And when I trained on 1670 step, it causes a mistake like
- INFO:tensorflow:global step 1640: loss = 0.8397 (0.360 sec/step)
- INFO:tensorflow:global step 1650: loss = 1.2242 (0.358 sec/step)
- INFO:tensorflow:global step 1660: loss = 0.9408 (0.352 sec/step)
- INFO:tensorflow:global step 1670: loss = 0.8304 (0.357 sec/step)
- INFO:tensorflow:global_step/sec: 2.79803
2018-08-10 09:16:21.108613: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Nan in summary histogram for: image_pooling/BatchNorm/moving_variance_1
[[Node: image_pooling/BatchNorm/moving_variance_1 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](image_pooling/BatchNorm/moving_variance_1/tag, image_pooling/BatchNorm/moving_variance/read)]]
2018-08-10 09:16:21.109451: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Nan in summary histogram for: image_pooling/BatchNorm/moving_variance_1
[[Node: image_pooling/BatchNorm/moving_variance_1 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](image_pooling/BatchNorm/moving_variance_1/tag, image_pooling/BatchNorm/moving_variance/read)]]
2018-08-10 09:16:21.109599: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Nan in summary histogram for: image_pooling/BatchNorm/moving_variance_1
[[Node: image_pooling/BatchNorm/moving_variance_1 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](image_pooling/BatchNorm/moving_variance_1/tag, image_pooling/BatchNorm/moving_variance/read)]]
2018-08-10 09:16:21.124892: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Nan in summary histogram for: image_pooling/BatchNorm/moving_variance_1
[[Node: image_pooling/BatchNorm/moving_variance_1 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](image_pooling/BatchNorm/moving_variance_1/tag, image_pooling/BatchNorm/moving_variance/read)]]
2018-08-10 09:16:21.125140: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Nan in summary histogram for: image_pooling/BatchNorm/moving_variance_1
[[Node: image_pooling/BatchNorm/moving_variance_1 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](image_pooling/BatchNorm/moving_variance_1/tag, image_pooling/BatchNorm/moving_variance/read)]]
`
Activity
Dammi87 commentedon Aug 10, 2018
How are you loading the model?
I had issues when I did it manually, but it turned out it was because I wasn't using the arg_scope to load it.
Here is how I load Resnet 152 (I'm using it as an Encoder in a VAE, so I needed to have nicer access to it)
Harshini-Gadige commentedon Sep 25, 2018
@SubWayss Hi, does this issue still exist ? If so, have you tried using arg_scope ?
wt-huang commentedon Nov 1, 2018
@Dammi87 is correct that arg_scope should work