python /home/admin/mtr/script_for_cron.py -j default -m 10 -a 'python3 ~/workarea/git/Velours/python/mtr/mask_rcnn/mask_rcnn_train.py --name=learn_mask_pancarte_200525_1 --hashtag_type=5007 -P 23147753 --class_name=badge,pancarte --steps_per_epoch=80 --lr=0.0005 ' -s learn_mask -M 31 -S 0 -U 100,100,120 2025-05-20 01:58:10.286209: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-20 01:58:10.315058: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-20 01:58:10.317152: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb724000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-20 01:58:10.317212: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-20 01:58:10.321200: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-20 01:58:10.480611: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3b42b670 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-20 01:58:10.480669: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-20 01:58:10.482156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-20 01:58:10.482624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-20 01:58:10.489548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-20 01:58:10.493455: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-20 01:58:10.493975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-20 01:58:10.497865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-20 01:58:10.499013: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-20 01:58:10.505881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-20 01:58:10.507438: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-20 01:58:10.507525: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-20 01:58:10.508370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-20 01:58:10.508384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-20 01:58:10.508393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-20 01:58:10.509714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_rcnn_train.py:33: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. model_param file didn't exist model_name : mask_coco_origin model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : [] file manque in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] local folder : /data/models_weight/mask_coco_origin import MySQLdb succeeded Import error (python version) warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.0005 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_mask_pancarte_200525_1 NUM_CLASSES 3 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 80 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 100 WEIGHT_DECAY 0.0001 launch train use defaut coco pre_trained model : /data/models_weight/mask_coco_origin/mask_model.h5 492993645 496005930 batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +{492993645: 'badge', 496005930: 'pancarte'} badge 492993645 1356331534 x0 : 639 y1 : 2294 width : 2076, height : 1311, area : 2721636, score : 1.0 None pancarte 496005930 1356331534 x0 : -1 y1 : 4158 width : 3125, height : 2735, area : 8546875, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331534.jpg pancarte 496005930 1356331538 x0 : 1319 y1 : 2643 width : 1206, height : 826, area : 996156, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331538.jpg pancarte 496005930 1356331545 x0 : 1109 y1 : 1843 width : 546, height : 431, area : 235326, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331545.jpg pancarte 496005930 1356331553 x0 : 971 y1 : 1646 width : 888, height : 679, area : 602952, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331553.jpg pancarte 496005930 1356331557 x0 : 1707 y1 : 1808 width : 774, height : 617, area : 477558, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331557.jpg pancarte 496005930 1356331559 x0 : 2057 y1 : 1619 width : 670, height : 514, area : 344380, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331559.jpg pancarte 496005930 1356331574 x0 : 0 y1 : 4159 width : 3119, height : 3332, area : 10392508, score : 1.0 None badge 492993645 1356331574 x0 : 772 y1 : 2309 width : 2341, height : 1582, area : 3703462, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331574.jpg pancarte 496005930 1356331578 x0 : 1710 y1 : 2479 width : 1107, height : 869, area : 961983, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331578.jpg pancarte 496005930 1356331586 x0 : 768 y1 : 2082 width : 840, height : 639, area : 536760, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331586.jpg pancarte 496005930 1356331593 x0 : 1150 y1 : 2168 width : 994, height : 818, area : 813092, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331593.jpg pancarte 496005930 1356331598 x0 : 1303 y1 : 2819 width : 1093, height : 982, area : 1073326, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331598.jpg pancarte 496005930 1356331602 x0 : 1052 y1 : 2716 width : 1124, height : 996, area : 1119504, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331602.jpg pancarte 496005930 1356331619 x0 : 2024 y1 : 2197 width : 725, height : 604, area : 437900, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331619.jpg badge 492993645 1356331626 x0 : 2158 y1 : 1396 width : 661, height : 455, area : 300755, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331626.jpg badge 492993645 1356331643 x0 : 1799 y1 : 1208 width : 976, height : 652, area : 636352, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331643.jpg pancarte 496005930 1356331654 x0 : 1386 y1 : 2579 width : 963, height : 832, area : 801216, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331654.jpg pancarte 496005930 1356331661 x0 : 1304 y1 : 2433 width : 896, height : 665, area : 595840, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331661.jpg badge 492993645 1356331664 x0 : 390 y1 : 3823 width : 2211, height : 3267, area : 7223337, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331664.jpg pancarte 496005930 1356331673 x0 : 1495 y1 : 1563 width : 1086, height : 723, area : 785178, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331673.jpg pancarte 496005930 1356331680 x0 : 2210 y1 : 1252 width : 302, height : 241, area : 72782, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331680.jpg pancarte 496005930 1356331687 x0 : 2909 y1 : 1573 width : 847, height : 607, area : 514129, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331687.jpg pancarte 496005930 1356331694 x0 : 2557 y1 : 1303 width : 360, height : 309, area : 111240, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331694.jpg pancarte 496005930 1356331701 x0 : 1174 y1 : 2191 width : 466, height : 316, area : 147256, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331701.jpg pancarte 496005930 1356331706 x0 : 696 y1 : 2860 width : 652, height : 526, area : 342952, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331706.jpg pancarte 496005930 1356331714 x0 : -1 y1 : 1164 width : 1367, height : 1164, area : 1591188, score : 1.0 None badge 492993645 1356331714 x0 : 0 y1 : 3010 width : 3124, height : 1839, area : 5745036, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331714.jpg pancarte 496005930 1356331719 x0 : 1962 y1 : 2387 width : 1408, height : 840, area : 1182720, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331719.jpg pancarte 496005930 1356331730 x0 : 1776 y1 : 1579 width : 666, height : 552, area : 367632, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331730.jpg pancarte 496005930 1356331735 x0 : 1135 y1 : 1927 width : 1195, height : 912, area : 1089840, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331735.jpg pancarte 496005930 1356331742 x0 : 2158 y1 : 1713 width : 950, height : 790, area : 750500, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331742.jpg pancarte 496005930 1356331746 x0 : 1992 y1 : 1564 width : 1138, height : 945, area : 1075410, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331746.jpg pancarte 496005930 1356331757 x0 : 1927 y1 : 1204 width : 823, height : 659, area : 542357, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331757.jpg pancarte 496005930 1356331762 x0 : 2647 y1 : 1081 width : 1122, height : 836, area : 937992, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331762.jpg pancarte 496005930 1356331766 x0 : 0 y1 : 3120 width : 4164, height : 3120, area : 12991680, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331766.jpg badge 492993645 1356331771 x0 : 0 y1 : 4159 width : 3122, height : 2498, area : 7798756, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331771.jpg pancarte 496005930 1356331779 x0 : 2154 y1 : 2091 width : 690, height : 632, area : 436080, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331779.jpg pancarte 496005930 1356331782 x0 : 1299 y1 : 1413 width : 1255, height : 826, area : 1036630, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331782.jpg pancarte 496005930 1356331788 x0 : 1995 y1 : 2575 width : 1505, height : 952, area : 1432760, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331788.jpg pancarte 496005930 1356331794 x0 : 1898 y1 : 2201 width : 793, height : 693, area : 549549, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331794.jpg pancarte 496005930 1356331801 x0 : 1475 y1 : 2996 width : 1319, height : 960, area : 1266240, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331801.jpg pancarte 496005930 1356331809 x0 : 2090 y1 : 1655 width : 614, height : 422, area : 259108, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331809.jpg pancarte 496005930 1356331815 x0 : 2211 y1 : 1627 width : 920, height : 670, area : 616400, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331815.jpg pancarte 496005930 1356331817 x0 : 1719 y1 : 2476 width : 795, height : 597, area : 474615, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331817.jpg badge 492993645 1356331838 x0 : 1 y1 : 3218 width : 3117, height : 2003, area : 6243351, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331838.jpg pancarte 496005930 1356331842 x0 : 1899 y1 : 2144 width : 1004, height : 771, area : 774084, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331842.jpg pancarte 496005930 1356331848 x0 : 2100 y1 : 2105 width : 1036, height : 806, area : 835016, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331848.jpg pancarte 496005930 1356331857 x0 : 2091 y1 : 2548 width : 1358, height : 999, area : 1356642, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331857.jpg pancarte 496005930 1356331862 x0 : 1590 y1 : 1690 width : 692, height : 396, area : 274032, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331862.jpg pancarte 496005930 1356331867 x0 : 1746 y1 : 1325 width : 845, height : 498, area : 420810, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331867.jpg pancarte 496005930 1356331876 x0 : 1609 y1 : 1724 width : 663, height : 474, area : 314262, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331876.jpg pancarte 496005930 1356331884 x0 : 1702 y1 : 2140 width : 1054, height : 767, area : 808418, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331884.jpg pancarte 496005930 1356331894 x0 : 172 y1 : 3120 width : 1632, height : 1175, area : 1917600, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331894.jpg pancarte 496005930 1356331903 x0 : 942 y1 : 2827 width : 989, height : 886, area : 876254, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331903.jpg pancarte 496005930 1356331907 x0 : 2192 y1 : 2257 width : 1537, height : 896, area : 1377152, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331907.jpg badge 492993645 1356331909 x0 : 1047 y1 : 2216 width : 3004, height : 1739, area : 5223956, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331909.jpg pancarte 496005930 1356331918 x0 : 1770 y1 : 2436 width : 632, height : 525, area : 331800, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331918.jpg pancarte 496005930 1356331923 x0 : 1454 y1 : 2979 width : 1160, height : 999, area : 1158840, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331923.jpg pancarte 496005930 1356331931 x0 : 1854 y1 : 1823 width : 484, height : 392, area : 189728, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331931.jpg pancarte 496005930 1356331943 x0 : 1826 y1 : 1571 width : 738, height : 572, area : 422136, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331943.jpg pancarte 496005930 1356331946 x0 : 2088 y1 : 1812 width : 782, height : 635, area : 496570, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331946.jpg pancarte 496005930 1356331957 x0 : 1804 y1 : 2287 width : 1318, height : 770, area : 1014860, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331957.jpg badge 492993645 1356331962 x0 : 9 y1 : 2615 width : 4151, height : 2608, area : 10825808, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331962.jpg pancarte 496005930 1356331967 x0 : 1452 y1 : 2869 width : 1218, height : 1081, area : 1316658, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331967.jpg pancarte 496005930 1356331976 x0 : 1331 y1 : 1960 width : 676, height : 511, area : 345436, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331976.jpg pancarte 496005930 1356331982 x0 : 1110 y1 : 2192 width : 693, height : 589, area : 408177, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331982.jpg pancarte 496005930 1356331985 x0 : 1999 y1 : 1817 width : 604, height : 480, area : 289920, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331985.jpg pancarte 496005930 1356331995 x0 : 1953 y1 : 2553 width : 1224, height : 817, area : 1000008, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356331995.jpg badge 492993645 1356332003 x0 : 2 y1 : 2685 width : 4149, height : 2685, area : 11140065, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356332003.jpg pancarte 496005930 1356332010 x0 : 2111 y1 : 2391 width : 861, height : 606, area : 521766, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356332010.jpg pancarte 496005930 1356332017 x0 : 2273 y1 : 2433 width : 939, height : 678, area : 636642, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356332017.jpg pancarte 496005930 1356332023 x0 : 1082 y1 : 1448 width : 366, height : 283, area : 103578, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356332023.jpg pancarte 496005930 1356332025 x0 : 2597 y1 : 2307 width : 504, height : 1056, area : 532224, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356332025.jpg pancarte 496005930 1356332042 x0 : 1872 y1 : 2500 width : 771, height : 549, area : 423279, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356332042.jpg pancarte 496005930 1356361517 x0 : 2274 y1 : 2363 width : 1381, height : 872, area : 1204232, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356361517.jpg badge 492993645 1356361523 x0 : 2010 y1 : 1624 width : 712, height : 445, area : 316840, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356361523.jpg pancarte 496005930 1356361530 x0 : 1595 y1 : 2256 width : 729, height : 597, area : 435213, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356361530.jpg pancarte 496005930 1356361537 x0 : 1900 y1 : 2125 width : 903, height : 724, area : 653772, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356361537.jpg pancarte 496005930 1356361543 x0 : 2208 y1 : 2389 width : 1015, height : 858, area : 870870, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356361543.jpg pancarte 496005930 1356361548 x0 : 1707 y1 : 2804 width : 1011, height : 880, area : 889680, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356361548.jpg pancarte 496005930 1356982967 x0 : 1773 y1 : 2444 width : 1347, height : 815, area : 1097805, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356982967.jpg badge 492993645 1356982971 x0 : 710 y1 : 4147 width : 2395, height : 3474, area : 8320230, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356982971.jpg pancarte 496005930 1356982975 x0 : 2092 y1 : 2226 width : 630, height : 502, area : 316260, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356982975.jpg pancarte 496005930 1356982983 x0 : 2134 y1 : 2335 width : 850, height : 675, area : 573750, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356982983.jpg pancarte 496005930 1356982991 x0 : 2351 y1 : 2008 width : 659, height : 532, area : 350588, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356982991.jpg pancarte 496005930 1356982997 x0 : 1309 y1 : 2176 width : 682, height : 510, area : 347820, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356982997.jpg pancarte 496005930 1356983040 x0 : 2101 y1 : 2457 width : 1047, height : 834, area : 873198, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356983040.jpg pancarte 496005930 1356983046 x0 : 1961 y1 : 2890 width : 1298, height : 1056, area : 1370688, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356983046.jpg badge 492993645 1356983052 x0 : 697 y1 : 2314 width : 2560, height : 1594, area : 4080640, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356983052.jpg pancarte 496005930 1356983061 x0 : 1541 y1 : 2587 width : 717, height : 552, area : 395784, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356983061.jpg pancarte 496005930 1356983067 x0 : 1264 y1 : 2104 width : 720, height : 543, area : 390960, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356983067.jpg pancarte 496005930 1356983071 x0 : 1181 y1 : 1546 width : 441, height : 298, area : 131418, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356983071.jpg pancarte 496005930 1356983086 x0 : 1603 y1 : 2607 width : 739, height : 540, area : 399060, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356983086.jpg pancarte 496005930 1356998184 x0 : 2033 y1 : 2402 width : 1383, height : 930, area : 1286190, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998184.jpg pancarte 496005930 1356998222 x0 : 2246 y1 : 2293 width : 635, height : 531, area : 337185, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998222.jpg pancarte 496005930 1356998225 x0 : 1619 y1 : 3994 width : 1188, height : 897, area : 1065636, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998225.jpg pancarte 496005930 1356998228 x0 : 1872 y1 : 2231 width : 671, height : 483, area : 324093, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998228.jpg pancarte 496005930 1356998234 x0 : 1252 y1 : 2346 width : 537, height : 290, area : 155730, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998234.jpg pancarte 496005930 1356998239 x0 : 1994 y1 : 1998 width : 515, height : 395, area : 203425, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998239.jpg pancarte 496005930 1356998244 x0 : 932 y1 : 2481 width : 665, height : 519, area : 345135, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998244.jpg pancarte 496005930 1356998320 x0 : 1577 y1 : 1513 width : 269, height : 241, area : 64829, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998320.jpg pancarte 496005930 1356998437 x0 : 1634 y1 : 3112 width : 950, height : 852, area : 809400, score : 1.0 None pancarte 496005930 1356998437 x0 : 1633 y1 : 3108 width : 945, height : 849, area : 802305, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998437.jpg pancarte 496005930 1356998545 x0 : 1214 y1 : 1860 width : 401, height : 363, area : 145563, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998545.jpg pancarte 496005930 1356998665 x0 : 1165 y1 : 2263 width : 904, height : 650, area : 587600, score : 1.0 None Using TensorFlow backend. WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1659: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. 2025-05-20 01:59:30.841938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-20 01:59:30.842036: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-20 01:59:30.842060: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-20 01:59:30.842084: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-20 01:59:30.842103: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-20 01:59:30.842123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-20 01:59:30.842141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-20 01:59:30.842162: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-20 01:59:30.843756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:570: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead 2025-05-20 01:59:36.398384: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 01:59:36.422603: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1363] Profiler found 1 GPUs 2025-05-20 01:59:36.426018: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcupti.so.10.1 2025-05-20 01:59:36.495931: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 01:59:54.090974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-20 01:59:54.091093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-20 01:59:54.091123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-20 01:59:54.091148: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-20 01:59:54.091169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-20 01:59:54.091193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-20 01:59:54.091215: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-20 01:59:54.091236: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-20 01:59:54.092656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-20 01:59:54.092717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-20 01:59:54.092732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-20 01:59:54.092744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-20 01:59:54.094260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/engine/training_generator.py:48: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class. warnings.warn( WARNING:tensorflow:Model failed to serialize as JSON. Ignoring... cannot pickle '_thread.RLock' object /data/data_root/learn_mask_pancarte_200525_1/train/1356998665.jpg badge 492993645 1356998732 x0 : 409 y1 : 4159 width : 2665, height : 4065, area : 10833225, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/train/1356998732.jpg pancarte 496005930 1356998753 x0 : 2109 y1 : 2432 width : 1295, height : 864, area : 1118880, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1356998753.jpg pancarte 496005930 1356998806 x0 : 1661 y1 : 1549 width : 608, height : 523, area : 317984, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1356998806.jpg pancarte 496005930 1356998813 x0 : 899 y1 : 1307 width : 1136, height : 1121, area : 1273456, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1356998813.jpg pancarte 496005930 1356998821 x0 : 1208 y1 : 1234 width : 482, height : 388, area : 187016, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1356998821.jpg pancarte 496005930 1356998826 x0 : 1050 y1 : 1365 width : 389, height : 322, area : 125258, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1356998826.jpg pancarte 496005930 1356998832 x0 : 1729 y1 : 1676 width : 470, height : 411, area : 193170, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1356998832.jpg pancarte 496005930 1357676318 x0 : 1942 y1 : 2041 width : 1334, height : 816, area : 1088544, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1357676318.jpg pancarte 496005930 1357676324 x0 : 1992 y1 : 2636 width : 928, height : 643, area : 596704, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1357676324.jpg pancarte 496005930 1357676331 x0 : 1758 y1 : 2800 width : 1140, height : 797, area : 908580, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1357676331.jpg pancarte 496005930 1357676341 x0 : 1756 y1 : 2940 width : 888, height : 948, area : 841824, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1357676341.jpg pancarte 496005930 1357676347 x0 : 1714 y1 : 3069 width : 1170, height : 1037, area : 1213290, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1357676347.jpg pancarte 496005930 1357676358 x0 : 1940 y1 : 2721 width : 882, height : 676, area : 596232, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_1/val/1357676358.jpg number of images :103 number of images :12 Training network heads Starting at epoch 0. LR=0.0005 Checkpoint Path: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/mask_rcnn_learn_mask_pancarte_200525_1_{epoch:04d}.h5 Selecting layers to train fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) 2025-05-20 02:00:32.014620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-20 02:00:32.874142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 Epoch 1/2 1/80 [..............................] - ETA: 23:05 - loss: 2.7439 - rpn_class_loss: 0.1183 - rpn_bbox_loss: 2.4981 - mrcnn_class_loss: 0.1275 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+002025-05-20 02:00:39.662169: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/80 [..............................] - ETA: 11:41 - loss: 2.6851 - rpn_class_loss: 0.1179 - rpn_bbox_loss: 2.4596 - mrcnn_class_loss: 0.1076 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+002025-05-20 02:00:40.101476: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:40.108925: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3279 callback api events and 3279 activity events. 2025-05-20 02:00:40.532770: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_40 2025-05-20 02:00:40.629859: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_40/marlene.trace.json.gz 2025-05-20 02:00:40.704805: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.272 ms 2025-05-20 02:00:40.736211: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_40Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_40/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_40/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_40/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_40/marlene.kernel_stats.pb 3/80 [>.............................] - ETA: 8:03 - loss: 3.0612 - rpn_class_loss: 0.0788 - rpn_bbox_loss: 2.0104 - mrcnn_class_loss: 0.1192 - mrcnn_bbox_loss: 0.4120 - mrcnn_mask_loss: 0.4408 2025-05-20 02:00:40.959931: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/80 [>.............................] - ETA: 6:02 - loss: 3.1585 - rpn_class_loss: 0.0597 - rpn_bbox_loss: 1.7784 - mrcnn_class_loss: 0.1307 - mrcnn_bbox_loss: 0.5248 - mrcnn_mask_loss: 0.66492025-05-20 02:00:41.184159: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:41.184785: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3462 callback api events and 3462 activity events. 2025-05-20 02:00:41.317228: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41 2025-05-20 02:00:41.384070: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.trace.json.gz 2025-05-20 02:00:41.429296: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.953 ms 2025-05-20 02:00:41.451302: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.kernel_stats.pb 5/80 [>.............................] - ETA: 4:53 - loss: 3.5804 - rpn_class_loss: 0.0562 - rpn_bbox_loss: 1.5503 - mrcnn_class_loss: 0.1503 - mrcnn_bbox_loss: 0.5785 - mrcnn_mask_loss: 1.24512025-05-20 02:00:41.681511: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/80 [=>............................] - ETA: 4:04 - loss: 3.8335 - rpn_class_loss: 0.0539 - rpn_bbox_loss: 1.3943 - mrcnn_class_loss: 0.1637 - mrcnn_bbox_loss: 0.6023 - mrcnn_mask_loss: 1.61932025-05-20 02:00:41.907842: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:41.908671: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:00:42.028637: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41 2025-05-20 02:00:42.097920: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.trace.json.gz 2025-05-20 02:00:42.142511: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.359 ms 2025-05-20 02:00:42.167308: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_41/marlene.kernel_stats.pb 7/80 [=>............................] - ETA: 3:31 - loss: 3.8443 - rpn_class_loss: 0.0466 - rpn_bbox_loss: 1.3675 - mrcnn_class_loss: 0.1908 - mrcnn_bbox_loss: 0.6589 - mrcnn_mask_loss: 1.58052025-05-20 02:00:42.427886: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/80 [==>...........................] - ETA: 3:04 - loss: 3.8279 - rpn_class_loss: 0.0411 - rpn_bbox_loss: 1.3456 - mrcnn_class_loss: 0.2065 - mrcnn_bbox_loss: 0.6941 - mrcnn_mask_loss: 1.54072025-05-20 02:00:42.660220: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:42.660906: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3466 callback api events and 3466 activity events. 2025-05-20 02:00:42.785986: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_42 2025-05-20 02:00:42.880788: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_42/marlene.trace.json.gz 2025-05-20 02:00:42.938159: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.732 ms 2025-05-20 02:00:42.970404: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_42Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_42/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_42/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_42/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_42/marlene.kernel_stats.pb 9/80 [==>...........................] - ETA: 2:59 - loss: 4.0188 - rpn_class_loss: 0.0366 - rpn_bbox_loss: 1.3628 - mrcnn_class_loss: 0.1890 - mrcnn_bbox_loss: 0.8549 - mrcnn_mask_loss: 1.57542025-05-20 02:00:44.805972: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/80 [==>...........................] - ETA: 2:48 - loss: 4.1569 - rpn_class_loss: 0.0330 - rpn_bbox_loss: 1.3723 - mrcnn_class_loss: 0.1755 - mrcnn_bbox_loss: 0.9716 - mrcnn_mask_loss: 1.60442025-05-20 02:00:46.189566: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:46.190586: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3463 activity events. 2025-05-20 02:00:46.390192: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_46 2025-05-20 02:00:46.474727: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_46/marlene.trace.json.gz 2025-05-20 02:00:46.521522: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.007 ms 2025-05-20 02:00:46.554962: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_46Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_46/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_46/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_46/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_46/marlene.kernel_stats.pb 11/80 [===>..........................] - ETA: 2:42 - loss: 4.1143 - rpn_class_loss: 0.0414 - rpn_bbox_loss: 1.3411 - mrcnn_class_loss: 0.1609 - mrcnn_bbox_loss: 0.9504 - mrcnn_mask_loss: 1.62052025-05-20 02:00:47.962998: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/80 [===>..........................] - ETA: 2:31 - loss: 4.0387 - rpn_class_loss: 0.0460 - rpn_bbox_loss: 1.2884 - mrcnn_class_loss: 0.1532 - mrcnn_bbox_loss: 0.9482 - mrcnn_mask_loss: 1.60292025-05-20 02:00:48.850665: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:48.851397: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3348 callback api events and 3348 activity events. 2025-05-20 02:00:49.007923: I $mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC==>.........................] - ETA: 2:14 - loss: 3.5524 - rpn_class_loss: 0.0624 - rpn_bbox_loss: 1.1848 - mrcnn_class_loss: 0.1310 - mrcnn_bbox_loss: 0.7950 - mrcnn_mask_loss: 1.37932025-05-20 02:00:53.231090: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/80 [=====>........................] - ETA: 2:08 - loss: 3.4824 - rpn_class_loss: 0.0615 - rpn_bbox_loss: 1.1508 - mrcnn_class_loss: 0.1317 - mrcnn_bbox_loss: 0.7738 - mrcnn_mask_loss: 1.36452025-05-20 02:00:54.322112: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:54.322968: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:00:54.482109: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_54 2025-05-20 02:00:54.543664: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_54/marlene.trace.json.gz 2025-05-20 02:00:54.589148: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.94 ms 2025-05-20 02:00:54.619376: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_54Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_54/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_54/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_54/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_54/marlene.kernel_stats.pb 17/80 [=====>........................] - ETA: 2:03 - loss: 3.4357 - rpn_class_loss: 0.0659 - rpn_bbox_loss: 1.0928 - mrcnn_class_loss: 0.1271 - mrcnn_bbox_loss: 0.7672 - mrcnn_mask_loss: 1.38272025-05-20 02:00:55.564040: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/80 [=====>........................] - ETA: 1:59 - loss: 3.3586 - rpn_class_loss: 0.0687 - rpn_bbox_loss: 1.0390 - mrcnn_class_loss: 0.1207 - mrcnn_bbox_loss: 0.7451 - mrcnn_mask_loss: 1.38512025-05-20 02:00:56.809093: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:56.809633: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3348 callback api events and 3348 activity events. 2025-05-20 02:00:56.941289: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_56 2025-05-20 02:00:57.006731: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_56/marlene.trace.json.gz 2025-05-20 02:00:57.057493: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.972 ms 2025-05-20 02:00:57.081391: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_56Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_56/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_56/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_56/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_56/marlene.kernel_stats.pb 19/80 [======>.......................] - ETA: 1:54 - loss: 3.3274 - rpn_class_loss: 0.0657 - rpn_bbox_loss: 1.0250 - mrcnn_class_loss: 0.1191 - mrcnn_bbox_loss: 0.7565 - mrcnn_mask_loss: 1.36112025-05-20 02:00:57.772185: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/80 [======>.......................] - ETA: 1:52 - loss: 3.2949 - rpn_class_loss: 0.0631 - rpn_bbox_loss: 1.0125 - mrcnn_class_loss: 0.1175 - mrcnn_bbox_loss: 0.7628 - mrcnn_mask_loss: 1.33902025-05-20 02:00:59.484676: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:00:59.485274: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3462 callback api events and 3462 activity events. 2025-05-20 02:00:59.641355: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_59 2025-05-20 02:00:59.719962: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_59/marlene.trace.json.gz 2025-05-20 02:00:59.767516: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.973 ms 2025-05-20 02:00:59.799798: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_59Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_59/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_59/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_59/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_00_59/marlene.kernel_stats.pb 21/80 [======>.......................] - ETA: 1:48 - loss: 3.2401 - rpn_class_loss: 0.0624 - rpn_bbox_loss: 0.9742 - mrcnn_class_loss: 0.1146 - mrcnn_bbox_loss: 0.7696 - mrcnn_mask_loss: 1.31932025-05-20 02:01:00.584453: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/80 [=======>......................] - ETA: 1:45 - loss: 3.1834 - rpn_class_loss: 0.0616 - rpn_bbox_loss: 0.9396 - mrcnn_class_loss: 0.1114 - mrcnn_bbox_loss: 0.7702 - mrcnn_mask_loss: 1.30062025-05-20 02:01:02.229836: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:02.230452: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3385 callback api events and 3385 activity events. 2025-05-20 02:01:02.432781: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_02 2025-05-20 02:01:02.527671: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_02/marlene.trace.json.gz 2025-05-20 02:01:02.582159: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.362 ms 2025-05-20 02:01:02.616022: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_02Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_02/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_02/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_02/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_02/marlene.kernel_stats.pb 23/80 [=======>......................] - ETA: 1:43 - loss: 3.1235 - rpn_class_loss: 0.0609 - rpn_bbox_loss: 0.9752 - mrcnn_class_loss: 0.1066 - mrcnn_bbox_loss: 0.7368 - mrcnn_mask_loss: 1.24402025-05-20 02:01:03.730067: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/80 [========>.....................] - ETA: 1:39 - loss: 3.0681 - rpn_class_loss: 0.0605 - rpn_bbox_loss: 1.0071 - mrcnn_class_loss: 0.1022 - mrcnn_bbox_loss: 0.7061 - mrcnn_mask_loss: 1.19222025-05-20 02:01:04.728011: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:04.728768: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3279 callback api events and 3279 activity events. 2025-05-20 02:01:04.899095: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_04 2025-05-20 02:01:04.970459: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_04/marlene.trace.json.gz 2025-05-20 02:01:05.022592: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.976 ms 2025-05-20 02:01:05.050754: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_04Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_04/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_04/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_04/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_04/marlene.kernel_stats.pb 25/80 [========>.....................] - ETA: 1:37 - loss: 3.0384 - rpn_class_loss: 0.0597 - rpn_bbox_loss: 0.9948 - mrcnn_class_loss: 0.0995 - mrcnn_bbox_loss: 0.7077 - mrcnn_mask_loss: 1.17672025-05-20 02:01:06.263861: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/80 [========>.....................] - ETA: 1:33 - loss: 3.0034 - rpn_class_loss: 0.0590 - rpn_bbox_loss: 0.9821 - mrcnn_class_loss: 0.0970 - mrcnn_bbox_loss: 0.7038 - mrcnn_mask_loss: 1.16142025-05-20 02:01:07.157851: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:07.158534: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3423 callback api events and 3423 activity events. 2025-05-20 02:01:07.328760: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_07 2025-05-20 02:01:07.394795: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_07/marlene.trace.json.gz 2025-05-20 02:01:07.454000: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.006 ms 2025-05-20 02:01:07.481514: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_07Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_07/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_07/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_07/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_07/marlene.kernel_stats.pb 27/80 [=========>....................] - ETA: 1:32 - loss: 2.9496 - rpn_class_loss: 0.0568 - rpn_bbox_loss: 0.9487 - mrcnn_class_loss: 0.0967 - mrcnn_bbox_loss: 0.7017 - mrcnn_mask_loss: 1.14572025-05-20 02:01:09.401605: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/80 [=========>....................] - ETA: 1:28 - loss: 2.9011 - rpn_class_loss: 0.0548 - rpn_bbox_loss: 0.9175 - mrcnn_class_loss: 0.0971 - mrcnn_bbox_loss: 0.7015 - mrcnn_mask_loss: 1.13022025-05-20 02:01:09.653926: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:09.654575: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:01:09.784789: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_09 2025-05-20 02:01:09.853277: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_09/marlene.trace.json.gz 2025-05-20 02:01:09.916817: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.315 ms 2025-05-20 02:01:09.944360: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_09/marlene.kernel_stats.pb 29/80 [=========>....................] - ETA: 1:28 - loss: 2.8482 - rpn_class_loss: 0.0541 - rpn_bbox_loss: 0.9318 - mrcnn_class_loss: 0.0937 - mrcnn_bbox_loss: 0.6773 - mrcnn_mask_loss: 1.09122025-05-20 02:01:12.472855: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/80 [==========>...................] - ETA: 1:24 - loss: 2.7843 - rpn_class_loss: 0.0534 - rpn_bbox_loss: 0.9307 - mrcnn_class_loss: 0.0906 - mrcnn_bbox_loss: 0.6547 - mrcnn_mask_loss: 1.05492025-05-20 02:01:12.750361: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:12.751713: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3279 callback api events and 3279 activity events. 2025-05-20 02:01:12.938724: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_12 2025-05-20 02:01:13.046875: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_12/marlene.trace.json.gz 2025-05-20 02:01:13.113498: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.402 ms 2025-05-20 02:01:13.149736: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_12/marlene.kernel_stats.pb 31/80 [==========>...................] - ETA: 1:25 - loss: 2.7608 - rpn_class_loss: 0.0519 - rpn_bbox_loss: 0.9275 - mrcnn_class_loss: 0.0898 - mrcnn_bbox_loss: 0.6501 - mrcnn_mask_loss: 1.04152025-05-20 02:01:16.106870: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/80 [===========>..................] - ETA: 1:21 - loss: 2.7359 - rpn_class_loss: 0.0504 - rpn_bbox_loss: 0.9237 - mrcnn_class_loss: 0.0882 - mrcnn_bbox_loss: 0.6446 - mrcnn_mask_loss: 1.02892025-05-20 02:01:16.398211: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:16.399246: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:01:16.559810: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_16 2025-05-20 02:01:16.660185: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_16/marlene.trace.json.gz 2025-05-20 02:01:16.720468: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.967 ms 2025-05-20 02:01:16.742074: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_16/marlene.kernel_stats.pb 33/80 [===========>..................] - ETA: 1:20 - loss: 2.6839 - rpn_class_loss: 0.0494 - rpn_bbox_loss: 0.9261 - mrcnn_class_loss: 0.0856 - mrcnn_bbox_loss: 0.6251 - mrcnn_mask_loss: 0.99772025-05-20 02:01:18.731860: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/80 [===========>..................] - ETA: 1:17 - loss: 2.6331 - rpn_class_loss: 0.0495 - rpn_bbox_loss: 0.9255 - mrcnn_class_loss: 0.0831 - mrcnn_bbox_loss: 0.6067 - mrcnn_mask_loss: 0.96842025-05-20 02:01:19.161057: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:19.162075: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3279 callback api events and 3279 activity events. 2025-05-20 02:01:19.359623: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_19 2025-05-20 02:01:19.458822: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_19/marlene.trace.json.gz 2025-05-20 02:01:19.538966: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.544 ms 2025-05-20 02:01:19.579524: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_19Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_19/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_19/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_19/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_19/marlene.kernel_stats.pb 35/80 [============>.................] - ETA: 1:18 - loss: 2.6281 - rpn_class_loss: 0.0494 - rpn_bbox_loss: 0.9216 - mrcnn_class_loss: 0.0820 - mrcnn_bbox_loss: 0.6170 - mrcnn_mask_loss: 0.95812025-05-20 02:01:23.176927: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/80 [============>.................] - ETA: 1:14 - loss: 2.6200 - rpn_class_loss: 0.0493 - rpn_bbox_loss: 0.9174 - mrcnn_class_loss: 0.0814 - mrcnn_bbox_loss: 0.6236 - mrcnn_mask_loss: 0.94832025-05-20 02:01:23.423389: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:23.424268: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3424 callback api events and 3424 activity events. 2025-05-20 02:01:23.632056: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_23 2025-05-20 02:01:23.730867: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_23/marlene.trace.json.gz 2025-05-20 02:01:23.811085: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.166 ms 2025-05-20 02:01:23.876168: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_23/marlene.kernel_stats.pb 37/80 [============>.................] - ETA: 1:12 - loss: 2.5928 - rpn_class_loss: 0.0487 - rpn_bbox_loss: 0.8959 - mrcnn_class_loss: 0.0802 - mrcnn_bbox_loss: 0.6278 - mrcnn_mask_loss: 0.94032025-05-20 02:01:24.786834: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/80 [=============>................] - ETA: 1:11 - loss: 2.5621 - rpn_class_loss: 0.0481 - rpn_bbox_loss: 0.8752 - mrcnn_class_loss: 0.0793 - mrcnn_bbox_loss: 0.6269 - mrcnn_mask_loss: 0.93262025-05-20 02:01:26.721378: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:26.723779: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3387 activity events. 2025-05-20 02:01:26.953838: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_26 2025-05-20 02:01:27.171581: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_26/marlene.trace.json.gz 2025-05-20 02:01:27.336977: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.534 ms 2025-05-20 02:01:27.388029: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_26/marlene.kernel_stats.pb 39/80 [=============>................] - ETA: 1:10 - loss: 2.5439 - rpn_class_loss: 0.0472 - rpn_bbox_loss: 0.8628 - mrcnn_class_loss: 0.0794 - mrcnn_bbox_loss: 0.6257 - mrcnn_mask_loss: 0.92872025-05-20 02:01:29.212295: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 40/80 [==============>...............] - ETA: 1:08 - loss: 2.5195 - rpn_class_loss: 0.0466 - rpn_bbox_loss: 0.8502 - mrcnn_class_loss: 0.0791 - mrcnn_bbox_loss: 0.6194 - mrcnn_mask_loss: 0.92422025-05-20 02:01:31.107641: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:31.108838: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:01:31.338450: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_31 2025-05-20 02:01:31.485784: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_31/marlene.trace.json.gz 2025-05-20 02:01:31.569381: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.524 ms 2025-05-20 02:01:31.611484: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_31Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_31/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_31/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_31/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_31/marlene.kernel_stats.pb 41/80 [==============>...............] - ETA: 1:08 - loss: 2.4822 - rpn_class_loss: 0.0464 - rpn_bbox_loss: 0.8311 - mrcnn_class_loss: 0.0782 - mrcnn_bbox_loss: 0.6102 - mrcnn_mask_loss: 0.91622025-05-20 02:01:34.559503: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 42/80 [==============>...............] - ETA: 1:09 - loss: 2.4463 - rpn_class_loss: 0.0460 - rpn_bbox_loss: 0.8136 - mrcnn_class_loss: 0.0775 - mrcnn_bbox_loss: 0.6014 - mrcnn_mask_loss: 0.90792025-05-20 02:01:39.218660: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:39.221150: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:01:39.798219: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_39 2025-05-20 02:01:39.936382: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_39/marlene.trace.json.gz 2025-05-20 02:01:40.023624: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.155 ms 2025-05-20 02:01:40.068484: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_39Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_39/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_39/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_39/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_39/marlene.kernel_stats.pb /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 43/80 [===============>..............] - ETA: 1:09 - loss: 2.4046 - rpn_class_loss: 0.0462 - rpn_bbox_loss: 0.8085 - mrcnn_class_loss: 0.0757 - mrcnn_bbox_loss: 0.5874 - mrcnn_mask_loss: 0.88682025-05-20 02:01:42.756742: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 44/80 [===============>..............] - ETA: 1:07 - loss: 2.3637 - rpn_class_loss: 0.0463 - rpn_bbox_loss: 0.8028 - mrcnn_class_loss: 0.0740 - mrcnn_bbox_loss: 0.5740 - mrcnn_mask_loss: 0.86672025-05-20 02:01:45.208855: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:45.209960: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3279 callback api events and 3279 activity events. 2025-05-20 02:01:45.481917: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_45 2025-05-20 02:01:45.589591: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_45/marlene.trace.json.gz 2025-05-20 02:01:45.719420: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.519 ms 2025-05-20 02:01:45.784498: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_45Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_45/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_45/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_45/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_45/marlene.kernel_stats.pb 45/80 [===============>..............] - ETA: 1:09 - loss: 2.3596 - rpn_class_loss: 0.0455 - rpn_bbox_loss: 0.7992 - mrcnn_class_loss: 0.0726 - mrcnn_bbox_loss: 0.5775 - mrcnn_mask_loss: 0.86482025-05-20 02:01:50.978777: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 46/80 [================>.............] - ETA: 1:11 - loss: 2.3544 - rpn_class_loss: 0.0448 - rpn_bbox_loss: 0.7952 - mrcnn_class_loss: 0.0716 - mrcnn_bbox_loss: 0.5801 - mrcnn_mask_loss: 0.86272025-05-20 02:01:58.996766: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:01:58.998527: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:01:59.376187: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_59 2025-05-20 02:01:59.511476: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_59/marlene.trace.json.gz 2025-05-20 02:01:59.557249: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.986 ms 2025-05-20 02:01:59.596405: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_59Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_59/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_59/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_59/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_01_59/marlene.kernel_stats.pb 47/80 [================>.............] - ETA: 1:09 - loss: 2.3372 - rpn_class_loss: 0.0438 - rpn_bbox_loss: 0.7881 - mrcnn_class_loss: 0.0717 - mrcnn_bbox_loss: 0.5794 - mrcnn_mask_loss: 0.85412025-05-20 02:02:00.650983: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 48/80 [=================>............] - ETA: 1:10 - loss: 2.3188 - rpn_class_loss: 0.0429 - rpn_bbox_loss: 0.7810 - mrcnn_class_loss: 0.0716 - mrcnn_bbox_loss: 0.5774 - mrcnn_mask_loss: 0.84592025-05-20 02:02:07.584638: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:02:07.585817: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3424 callback api events and 3424 activity events. 2025-05-20 02:02:07.800194: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_07 2025-05-20 02:02:07.889562: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_07/marlene.trace.json.gz 2025-05-20 02:02:07.958983: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.534 ms 2025-05-20 02:02:08.001177: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_07Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_07/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_07/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_07/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_07/marlene.kernel_stats.pb 49/80 [=================>............] - ETA: 1:12 - loss: 2.2977 - rpn_class_loss: 0.0423 - rpn_bbox_loss: 0.7669 - mrcnn_class_loss: 0.0717 - mrcnn_bbox_loss: 0.5761 - mrcnn_mask_loss: 0.84082025-05-20 02:02:16.734633: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 50/80 [=================>............] - ETA: 1:08 - loss: 2.2773 - rpn_class_loss: 0.0416 - rpn_bbox_loss: 0.7523 - mrcnn_class_loss: 0.0716 - mrcnn_bbox_loss: 0.5761 - mrcnn_mask_loss: 0.83572025-05-20 02:02:17.043723: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:02:17.044697: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3424 callback api events and 3424 activity events. 2025-05-20 02:02:17.280894: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_17 2025-05-20 02:02:17.372306: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_17/marlene.trace.json.gz 2025-05-20 02:02:17.463162: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.608 ms 2025-05-20 02:02:17.502507: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_17Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_17/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_17/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_17/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_17/marlene.kernel_stats.pb 51/80 [==================>...........] - ETA: 1:13 - loss: 2.2578 - rpn_class_loss: 0.0415 - rpn_bbox_loss: 0.7492 - mrcnn_class_loss: 0.0704 - mrcnn_bbox_loss: 0.5720 - mrcnn_mask_loss: 0.82482025-05-20 02:02:31.095259: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 52/80 [==================>...........] - ETA: 1:13 - loss: 2.2354 - rpn_class_loss: 0.0414 - rpn_bbox_loss: 0.7454 - mrcnn_class_loss: 0.0691 - mrcnn_bbox_loss: 0.5651 - mrcnn_mask_loss: 0.81432025-05-20 02:02:39.249138: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:02:39.250862: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3348 callback api events and 3348 activity events. 2025-05-20 02:02:39.473417: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_39 2025-05-20 02:02:39.605502: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_39/marlene.trace.json.gz 2025-05-20 02:02:39.703246: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.909 ms 2025-05-20 02:02:39.749079: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_39Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_39/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_39/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_39/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_02_39/marlene.kernel_stats.pb 53/80 [==================>...........] - ETA: 1:15 - loss: 2.2527 - rpn_class_loss: 0.0409 - rpn_bbox_loss: 0.7540 - mrcnn_class_loss: 0.0724 - mrcnn_bbox_loss: 0.5749 - mrcnn_mask_loss: 0.81052025-05-20 02:02:51.059270: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 54/80 [===================>..........] - ETA: 1:15 - loss: 2.2613 - rpn_class_loss: 0.0404 - rpn_bbox_loss: 0.7571 - mrcnn_class_loss: 0.0753 - mrcnn_bbox_loss: 0.5821 - mrcnn_mask_loss: 0.80652025-05-20 02:02:59.941618: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:02:59.943402: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3466 callback api events and 3466 activity events. 2025-05-20 02:03:00.191277: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_00 2025-05-20 02:03:00.305856: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_00/marlene.trace.json.gz 2025-05-20 02:03:00.396244: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.209 ms 2025-05-20 02:03:00.442115: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_00Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_00/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_00/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_00/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_00/marlene.kernel_stats.pb 55/80 [===================>..........] - ETA: 1:17 - loss: 2.2473 - rpn_class_loss: 0.0400 - rpn_bbox_loss: 0.7609 - mrcnn_class_loss: 0.0745 - mrcnn_bbox_loss: 0.5741 - mrcnn_mask_loss: 0.79772025-05-20 02:03:11.571209: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 56/80 [====================>.........] - ETA: 1:12 - loss: 2.2325 - rpn_class_loss: 0.0396 - rpn_bbox_loss: 0.7642 - mrcnn_class_loss: 0.0736 - mrcnn_bbox_loss: 0.5660 - mrcnn_mask_loss: 0.78912025-05-20 02:03:12.337288: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:03:12.338646: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3347 callback api events and 3347 activity events. 2025-05-20 02:03:12.699387: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_12 2025-05-20 02:03:12.837453: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_12/marlene.trace.json.gz 2025-05-20 02:03:13.034989: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.769 ms 2025-05-20 02:03:13.141636: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_12/marlene.kernel_stats.pb 57/80 [====================>.........] - ETA: 1:14 - loss: 2.2507 - rpn_class_loss: 0.0389 - rpn_bbox_loss: 0.7701 - mrcnn_class_loss: 0.0743 - mrcnn_bbox_loss: 0.5761 - mrcnn_mask_loss: 0.79132025-05-20 02:03:26.577078: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 58/80 [====================>.........] - ETA: 1:10 - loss: 2.2643 - rpn_class_loss: 0.0383 - rpn_bbox_loss: 0.7750 - mrcnn_class_loss: 0.0747 - mrcnn_bbox_loss: 0.5824 - mrcnn_mask_loss: 0.79392025-05-20 02:03:27.016339: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:03:27.017422: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3463 activity events. 2025-05-20 02:03:27.296434: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_27 2025-05-20 02:03:27.423349: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_27/marlene.trace.json.gz 2025-05-20 02:03:27.504566: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.63 ms 2025-05-20 02:03:27.561326: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_27Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_27/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_27/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_27/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_27/marlene.kernel_stats.pb 59/80 [=====================>........] - ETA: 1:09 - loss: 2.2628 - rpn_class_loss: 0.0379 - rpn_bbox_loss: 0.7771 - mrcnn_class_loss: 0.0739 - mrcnn_bbox_loss: 0.5852 - mrcnn_mask_loss: 0.78872025-05-20 02:03:36.895589: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 60/80 [=====================>........] - ETA: 1:05 - loss: 2.2569 - rpn_class_loss: 0.0377 - rpn_bbox_loss: 0.7775 - mrcnn_class_loss: 0.0735 - mrcnn_bbox_loss: 0.5837 - mrcnn_mask_loss: 0.78462025-05-20 02:03:39.486823: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:03:39.488960: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3387 activity events. 2025-05-20 02:03:39.907594: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_39 2025-05-20 02:03:40.089910: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_39/marlene.trace.json.gz 2025-05-20 02:03:40.199742: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.704 ms 2025-05-20 02:03:40.255095: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_39Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_39/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_39/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_39/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_03_39/marlene.kernel_stats.pb 61/80 [=====================>........] - ETA: 1:08 - loss: 2.2803 - rpn_class_loss: 0.0370 - rpn_bbox_loss: 0.7883 - mrcnn_class_loss: 0.0736 - mrcnn_bbox_loss: 0.5936 - mrcnn_mask_loss: 0.78782025-05-20 02:04:03.483013: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 62/80 [======================>.......] - ETA: 1:04 - loss: 2.2994 - rpn_class_loss: 0.0365 - rpn_bbox_loss: 0.7979 - mrcnn_class_loss: 0.0732 - mrcnn_bbox_loss: 0.6019 - mrcnn_mask_loss: 0.78992025-05-20 02:04:03.825279: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:04:03.828190: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3462 callback api events and 3462 activity events. 2025-05-20 02:04:04.196988: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_04 2025-05-20 02:04:04.346115: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_04/marlene.trace.json.gz 2025-05-20 02:04:04.453662: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.228 ms 2025-05-20 02:04:04.508578: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_04Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_04/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_04/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_04/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_04/marlene.kernel_stats.pb 63/80 [======================>.......] - ETA: 1:01 - loss: 2.2866 - rpn_class_loss: 0.0363 - rpn_bbox_loss: 0.7912 - mrcnn_class_loss: 0.0725 - mrcnn_bbox_loss: 0.6047 - mrcnn_mask_loss: 0.78182025-05-20 02:04:09.660866: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 64/80 [=======================>......] - ETA: 57s - loss: 2.2721 - rpn_class_loss: 0.0362 - rpn_bbox_loss: 0.7841 - mrcnn_class_loss: 0.0721 - mrcnn_bbox_loss: 0.6059 - mrcnn_mask_loss: 0.7739 2025-05-20 02:04:13.115088: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:04:13.116768: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3385 callback api events and 3385 activity events. 2025-05-20 02:04:13.420572: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_13 2025-05-20 02:04:13.489985: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_13/marlene.trace.json.gz 2025-05-20 02:04:13.562469: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.539 ms 2025-05-20 02:04:13.597764: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_13Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_13/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_13/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_13/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_13/marlene.kernel_stats.pb 65/80 [=======================>......] - ETA: 55s - loss: 2.2600 - rpn_class_loss: 0.0361 - rpn_bbox_loss: 0.7804 - mrcnn_class_loss: 0.0712 - mrcnn_bbox_loss: 0.6047 - mrcnn_mask_loss: 0.76762025-05-20 02:04:24.071890: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 66/80 [=======================>......] - ETA: 52s - loss: 2.2461 - rpn_class_loss: 0.0360 - rpn_bbox_loss: 0.7766 - mrcnn_class_loss: 0.0706 - mrcnn_bbox_loss: 0.6015 - mrcnn_mask_loss: 0.76152025-05-20 02:04:28.288187: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:04:28.290251: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3387 activity events. 2025-05-20 02:04:28.598871: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_28 2025-05-20 02:04:28.783077: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_28/marlene.trace.json.gz 2025-05-20 02:04:28.903561: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.662 ms 2025-05-20 02:04:28.974081: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_28Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_28/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_28/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_28/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_28/marlene.kernel_stats.pb 67/80 [========================>.....] - ETA: 49s - loss: 2.2323 - rpn_class_loss: 0.0356 - rpn_bbox_loss: 0.7742 - mrcnn_class_loss: 0.0700 - mrcnn_bbox_loss: 0.5961 - mrcnn_mask_loss: 0.75632025-05-20 02:04:39.783233: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 68/80 [========================>.....] - ETA: 46s - loss: 2.2179 - rpn_class_loss: 0.0353 - rpn_bbox_loss: 0.7716 - mrcnn_class_loss: 0.0696 - mrcnn_bbox_loss: 0.5901 - mrcnn_mask_loss: 0.75122025-05-20 02:04:43.237068: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:04:43.239228: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3387 activity events. 2025-05-20 02:04:43.549748: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_43 2025-05-20 02:04:43.696883: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_43/marlene.trace.json.gz 2025-05-20 02:04:43.807415: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.814 ms 2025-05-20 02:04:43.867744: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_43Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_43/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_43/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_43/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_04_43/marlene.kernel_stats.pb 69/80 [========================>.....] - ETA: 44s - loss: 2.2038 - rpn_class_loss: 0.0352 - rpn_bbox_loss: 0.7692 - mrcnn_class_loss: 0.0686 - mrcnn_bbox_loss: 0.5891 - mrcnn_mask_loss: 0.74172025-05-20 02:05:03.403696: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 70/80 [=========================>....] - ETA: 40s - loss: 2.1886 - rpn_class_loss: 0.0350 - rpn_bbox_loss: 0.7663 - mrcnn_class_loss: 0.0677 - mrcnn_bbox_loss: 0.5871 - mrcnn_mask_loss: 0.73242025-05-20 02:05:03.768419: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:05:03.770620: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:05:06.537381: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_06 2025-05-20 02:05:06.639583: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_06/marlene.trace.json.gz 2025-05-20 02:05:06.790531: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.753 ms 2025-05-20 02:05:06.869699: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_06Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_06/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_06/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_06/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_06/marlene.kernel_stats.pb 71/80 [=========================>....] - ETA: 37s - loss: 2.1850 - rpn_class_loss: 0.0348 - rpn_bbox_loss: 0.7631 - mrcnn_class_loss: 0.0669 - mrcnn_bbox_loss: 0.5914 - mrcnn_mask_loss: 0.72882025-05-20 02:05:14.427835: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 72/80 [==========================>...] - ETA: 33s - loss: 2.1804 - rpn_class_loss: 0.0346 - rpn_bbox_loss: 0.7596 - mrcnn_class_loss: 0.0661 - mrcnn_bbox_loss: 0.5947 - mrcnn_mask_loss: 0.72532025-05-20 02:05:22.256834: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:05:22.258510: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3389 callback api events and 3389 activity events. 2025-05-20 02:05:22.516521: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_22 2025-05-20 02:05:22.745916: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_22/marlene.trace.json.gz 2025-05-20 02:05:22.907443: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.491 ms 2025-05-20 02:05:23.007974: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_22Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_22/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_22/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_22/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_22/marlene.kernel_stats.pb 73/80 [==========================>...] - ETA: 29s - loss: 2.1827 - rpn_class_loss: 0.0342 - rpn_bbox_loss: 0.7549 - mrcnn_class_loss: 0.0669 - mrcnn_bbox_loss: 0.6047 - mrcnn_mask_loss: 0.72202025-05-20 02:05:27.614748: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 74/80 [==========================>...] - ETA: 25s - loss: 2.1835 - rpn_class_loss: 0.0338 - rpn_bbox_loss: 0.7491 - mrcnn_class_loss: 0.0676 - mrcnn_bbox_loss: 0.6139 - mrcnn_mask_loss: 0.71912025-05-20 02:05:34.834838: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:05:34.839204: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3463 activity events. 2025-05-20 02:05:35.196849: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_35 2025-05-20 02:05:35.343541: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_35/marlene.trace.json.gz 2025-05-20 02:05:35.421346: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.068 ms 2025-05-20 02:05:35.473562: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_35Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_35/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_35/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_35/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_35/marlene.kernel_stats.pb 75/80 [===========================>..] - ETA: 22s - loss: 2.1684 - rpn_class_loss: 0.0336 - rpn_bbox_loss: 0.7424 - mrcnn_class_loss: 0.0672 - mrcnn_bbox_loss: 0.6117 - mrcnn_mask_loss: 0.71342025-05-20 02:05:52.851803: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 76/80 [===========================>..] - ETA: 17s - loss: 2.1533 - rpn_class_loss: 0.0334 - rpn_bbox_loss: 0.7357 - mrcnn_class_loss: 0.0668 - mrcnn_bbox_loss: 0.6091 - mrcnn_mask_loss: 0.70832025-05-20 02:05:59.419850: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:05:59.430707: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3463 activity events. 2025-05-20 02:05:59.883759: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_59 2025-05-20 02:05:59.952843: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_59/marlene.trace.json.gz 2025-05-20 02:06:00.815977: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 18.886 ms 2025-05-20 02:06:00.957554: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_59Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_59/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_59/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_59/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_05_59/marlene.kernel_stats.pb 77/80 [===========================>..] - ETA: 13s - loss: 2.1422 - rpn_class_loss: 0.0332 - rpn_bbox_loss: 0.7327 - mrcnn_class_loss: 0.0664 - mrcnn_bbox_loss: 0.6055 - mrcnn_mask_loss: 0.70452025-05-20 02:06:08.844237: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 78/80 [============================>.] - ETA: 9s - loss: 2.1283 - rpn_class_loss: 0.0330 - rpn_bbox_loss: 0.7287 - mrcnn_class_loss: 0.0657 - mrcnn_bbox_loss: 0.6004 - mrcnn_mask_loss: 0.7004 2025-05-20 02:06:16.538400: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:06:16.540590: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:06:17.047849: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_16 2025-05-20 02:06:17.286113: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_16/marlene.trace.json.gz 2025-05-20 02:06:17.483652: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.996 ms 2025-05-20 02:06:17.587874: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_16/marlene.kernel_stats.pb 79/80 [============================>.] - ETA: 4s - loss: 2.1172 - rpn_class_loss: 0.0327 - rpn_bbox_loss: 0.7227 - mrcnn_class_loss: 0.0651 - mrcnn_bbox_loss: 0.6001 - mrcnn_mask_loss: 0.69652025-05-20 02:06:28.359379: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:06:34.438355: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:06:34.441900: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:06:35.307328: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_34 2025-05-20 02:06:35.662231: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_34/marlene.trace.json.gz 2025-05-20 02:06:35.954370: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.926 ms 2025-05-20 02:06:36.140825: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_06_34/marlene.kernel_stats.pb 80/80 [==============================] - 399s 5s/step - loss: 2.1052 - rpn_class_loss: 0.0324 - rpn_bbox_loss: 0.7166 - mrcnn_class_loss: 0.0645 - mrcnn_bbox_loss: 0.5988 - mrcnn_mask_loss: 0.6929 - val_loss: 2.3311 - val_rpn_class_loss: 0.0147 - val_rpn_bbox_loss: 1.4842 - val_mrcnn_class_loss: 0.0242 - val_mrcnn_bbox_loss: 0.4267 - val_mrcnn_mask_loss: 0.3814 /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Epoch 2/2 1/80 [..............................] - ETA: 22s - loss: 1.2719 - rpn_class_loss: 0.0183 - rpn_bbox_loss: 0.4165 - mrcnn_class_loss: 0.0364 - mrcnn_bbox_loss: 0.3991 - mrcnn_mask_loss: 0.40162025-05-20 02:11:05.491614: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/80 [..............................] - ETA: 35s - loss: 1.2223 - rpn_class_loss: 0.0183 - rpn_bbox_loss: 0.4102 - mrcnn_class_loss: 0.0408 - mrcnn_bbox_loss: 0.3544 - mrcnn_mask_loss: 0.39862025-05-20 02:11:06.311205: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:06.321870: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:11:06.909133: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_06 2025-05-20 02:11:07.054596: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_06/marlene.trace.json.gz 2025-05-20 02:11:07.272611: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.163 ms 2025-05-20 02:11:07.374160: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_06Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_06/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_06/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_06/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_06/marlene.kernel_stats.pb 3/80 [>.............................] - ETA: 1:05 - loss: 1.1178 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.3265 - mrcnn_class_loss: 0.0412 - mrcnn_bbox_loss: 0.3680 - mrcnn_mask_loss: 0.36612025-05-20 02:11:07.751791: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/80 [>.............................] - ETA: 58s - loss: 1.0458 - rpn_class_loss: 0.0153 - rpn_bbox_loss: 0.2722 - mrcnn_class_loss: 0.0344 - mrcnn_bbox_loss: 0.3581 - mrcnn_mask_loss: 0.3657 2025-05-20 02:11:08.420937: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:08.424843: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:11:09.484844: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_09 2025-05-20 02:11:09.651733: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_09/marlene.trace.json.gz 2025-05-20 02:11:09.760836: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.924 ms 2025-05-20 02:11:09.832827: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_09/marlene.kernel_stats.pb 5/80 [>.............................] - ETA: 1:16 - loss: 1.2339 - rpn_class_loss: 0.0168 - rpn_bbox_loss: 0.3694 - mrcnn_class_loss: 0.0334 - mrcnn_bbox_loss: 0.4344 - mrcnn_mask_loss: 0.37992025-05-20 02:11:10.310036: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/80 [=>............................] - ETA: 1:10 - loss: 1.3440 - rpn_class_loss: 0.0178 - rpn_bbox_loss: 0.4259 - mrcnn_class_loss: 0.0342 - mrcnn_bbox_loss: 0.4727 - mrcnn_mask_loss: 0.39342025-05-20 02:11:11.028122: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:11.036529: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3426 activity events. 2025-05-20 02:11:11.972941: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_11 2025-05-20 02:11:12.292411: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_11/marlene.trace.json.gz 2025-05-20 02:11:12.420774: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.029 ms 2025-05-20 02:11:12.476676: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_11Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_11/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_11/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_11/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_11/marlene.kernel_stats.pb 7/80 [=>............................] - ETA: 1:23 - loss: 1.3780 - rpn_class_loss: 0.0162 - rpn_bbox_loss: 0.4775 - mrcnn_class_loss: 0.0342 - mrcnn_bbox_loss: 0.4544 - mrcnn_mask_loss: 0.39572025-05-20 02:11:13.179810: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/80 [==>...........................] - ETA: 1:16 - loss: 1.3967 - rpn_class_loss: 0.0152 - rpn_bbox_loss: 0.5227 - mrcnn_class_loss: 0.0315 - mrcnn_bbox_loss: 0.4338 - mrcnn_mask_loss: 0.39352025-05-20 02:11:13.905745: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:13.910795: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3424 callback api events and 3424 activity events. 2025-05-20 02:11:14.382415: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_14 2025-05-20 02:11:14.534523: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_14/marlene.trace.json.gz 2025-05-20 02:11:14.667192: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.128 ms 2025-05-20 02:11:14.803249: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_14Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_14/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_14/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_14/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_14/marlene.kernel_stats.pb 9/80 [==>...........................] - ETA: 1:20 - loss: 1.3794 - rpn_class_loss: 0.0184 - rpn_bbox_loss: 0.5486 - mrcnn_class_loss: 0.0309 - mrcnn_bbox_loss: 0.4047 - mrcnn_mask_loss: 0.37682025-05-20 02:11:15.398879: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/80 [==>...........................] - ETA: 1:14 - loss: 1.3555 - rpn_class_loss: 0.0209 - rpn_bbox_loss: 0.5644 - mrcnn_class_loss: 0.0309 - mrcnn_bbox_loss: 0.3768 - mrcnn_mask_loss: 0.36252025-05-20 02:11:15.870384: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:15.871744: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3387 activity events. 2025-05-20 02:11:16.370867: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_16 2025-05-20 02:11:16.458106: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_16/marlene.trace.json.gz 2025-05-20 02:11:16.532572: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.878 ms 2025-05-20 02:11:16.593473: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_16/marlene.kernel_stats.pb 11/80 [===>..........................] - ETA: 1:14 - loss: 1.3660 - rpn_class_loss: 0.0204 - rpn_bbox_loss: 0.5486 - mrcnn_class_loss: 0.0293 - mrcnn_bbox_loss: 0.3989 - mrcnn_mask_loss: 0.36882025-05-20 02:11:17.011262: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/80 [===>..........................] - ETA: 1:09 - loss: 1.3577 - rpn_class_loss: 0.0203 - rpn_bbox_loss: 0.5338 - mrcnn_class_loss: 0.0282 - mrcnn_bbox_loss: 0.4078 - mrcnn_mask_loss: 0.36752025-05-20 02:11:17.455609: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:17.456993: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3423 callback api events and 3423 activity events. 2025-05-20 02:11:18.061487: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_17 2025-05-20 02:11:18.198652: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_17/marlene.trace.json.gz 2025-05-20 02:11:18.301321: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.701 ms 2025-05-20 02:11:18.362290: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_17Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_17/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_17/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_17/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_17/marlene.kernel_stats.pb 13/80 [===>..........................] - ETA: 1:10 - loss: 1.2878 - rpn_class_loss: 0.0208 - rpn_bbox_loss: 0.5016 - mrcnn_class_loss: 0.0262 - mrcnn_bbox_loss: 0.3856 - mrcnn_mask_loss: 0.35362025-05-20 02:11:18.853047: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/80 [====>.........................] - ETA: 1:07 - loss: 1.2566 - rpn_class_loss: 0.0212 - rpn_bbox_loss: 0.4744 - mrcnn_class_loss: 0.0246 - mrcnn_bbox_loss: 0.3843 - mrcnn_mask_loss: 0.35212025-05-20 02:11:19.752801: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:19.754658: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3387 activity events. 2025-05-20 02:11:20.258749: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_20 2025-05-20 02:11:20.438982: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_20/marlene.trace.json.gz 2025-05-20 02:11:20.703874: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.331 ms 2025-05-20 02:11:20.857115: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_20Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_20/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_20/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_20/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_20/marlene.kernel_stats.pb 15/80 [====>.........................] - ETA: 1:09 - loss: 1.2946 - rpn_class_loss: 0.0203 - rpn_bbox_loss: 0.4632 - mrcnn_class_loss: 0.0281 - mrcnn_bbox_loss: 0.4019 - mrcnn_mask_loss: 0.38102025-05-20 02:11:21.256859: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/80 [=====>........................] - ETA: 1:06 - loss: 1.3210 - rpn_class_loss: 0.0196 - rpn_bbox_loss: 0.4521 - mrcnn_class_loss: 0.0309 - mrcnn_bbox_loss: 0.4125 - mrcnn_mask_loss: 0.40592025-05-20 02:11:21.792723: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:21.793936: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3462 callback api events and 3463 activity events. 2025-05-20 02:11:22.138093: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_21 2025-05-20 02:11:22.387314: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_21/marlene.trace.json.gz 2025-05-20 02:11:22.768734: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.216 ms 2025-05-20 02:11:23.066091: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_21Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_21/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_21/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_21/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_21/marlene.kernel_stats.pb 17/80 [=====>........................] - ETA: 1:08 - loss: 1.3266 - rpn_class_loss: 0.0187 - rpn_bbox_loss: 0.4702 - mrcnn_class_loss: 0.0299 - mrcnn_bbox_loss: 0.4136 - mrcnn_mask_loss: 0.39412025-05-20 02:11:23.730095: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/80 [=====>........................] - ETA: 1:05 - loss: 1.3271 - rpn_class_loss: 0.0178 - rpn_bbox_loss: 0.4843 - mrcnn_class_loss: 0.0286 - mrcnn_bbox_loss: 0.4127 - mrcnn_mask_loss: 0.38372025-05-20 02:11:24.194475: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:24.195344: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:11:24.494811: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_24 2025-05-20 02:11:24.664684: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_24/marlene.trace.json.gz 2025-05-20 02:11:24.923697: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.61 ms 2025-05-20 02:11:24.999435: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_24Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_24/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_24/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_24/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_24/marlene.kernel_stats.pb 19/80 [======>.......................] - ETA: 1:05 - loss: 1.3133 - rpn_class_loss: 0.0175 - rpn_bbox_loss: 0.4794 - mrcnn_class_loss: 0.0276 - mrcnn_bbox_loss: 0.4108 - mrcnn_mask_loss: 0.37802025-05-20 02:11:25.546455: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/80 [======>.......................] - ETA: 1:03 - loss: 1.2936 - rpn_class_loss: 0.0172 - rpn_bbox_loss: 0.4644 - mrcnn_class_loss: 0.0273 - mrcnn_bbox_loss: 0.4112 - mrcnn_mask_loss: 0.37352025-05-20 02:11:26.327206: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:26.331470: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:11:27.104472: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_26 2025-05-20 02:11:27.361181: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_26/marlene.trace.json.gz 2025-05-20 02:11:27.461787: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.238 ms 2025-05-20 02:11:27.566769: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_26/marlene.kernel_stats.pb 21/80 [======>.......................] - ETA: 1:04 - loss: 1.3263 - rpn_class_loss: 0.0171 - rpn_bbox_loss: 0.5123 - mrcnn_class_loss: 0.0273 - mrcnn_bbox_loss: 0.4021 - mrcnn_mask_loss: 0.36752025-05-20 02:11:28.159439: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/80 [=======>......................] - ETA: 1:01 - loss: 1.3367 - rpn_class_loss: 0.0166 - rpn_bbox_loss: 0.5400 - mrcnn_class_loss: 0.0273 - mrcnn_bbox_loss: 0.3915 - mrcnn_mask_loss: 0.36132025-05-20 02:11:28.986355: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:29.001178: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3464 activity events. 2025-05-20 02:11:30.205444: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_29 2025-05-20 02:11:30.333794: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_29/marlene.trace.json.gz 2025-05-20 02:11:30.456496: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.262 ms 2025-05-20 02:11:30.557233: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_29Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_29/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_29/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_29/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_29/marlene.kernel_stats.pb 23/80 [=======>......................] - ETA: 1:09 - loss: 1.3278 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.5408 - mrcnn_class_loss: 0.0279 - mrcnn_bbox_loss: 0.3813 - mrcnn_mask_loss: 0.36162025-05-20 02:11:33.368363: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/80 [========>.....................] - ETA: 1:08 - loss: 1.3198 - rpn_class_loss: 0.0158 - rpn_bbox_loss: 0.5404 - mrcnn_class_loss: 0.0291 - mrcnn_bbox_loss: 0.3716 - mrcnn_mask_loss: 0.36292025-05-20 02:11:34.522514: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:34.524891: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3426 activity events. 2025-05-20 02:11:34.801332: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_34 2025-05-20 02:11:35.125572: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_34/marlene.trace.json.gz 2025-05-20 02:11:35.315033: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.897 ms 2025-05-20 02:11:35.387177: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_34/marlene.kernel_stats.pb 25/80 [========>.....................] - ETA: 1:14 - loss: 1.2913 - rpn_class_loss: 0.0157 - rpn_bbox_loss: 0.5255 - mrcnn_class_loss: 0.0287 - mrcnn_bbox_loss: 0.3622 - mrcnn_mask_loss: 0.35942025-05-20 02:11:38.849376: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/80 [========>.....................] - ETA: 1:10 - loss: 1.2649 - rpn_class_loss: 0.0155 - rpn_bbox_loss: 0.5122 - mrcnn_class_loss: 0.0286 - mrcnn_bbox_loss: 0.3536 - mrcnn_mask_loss: 0.35512025-05-20 02:11:39.123539: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:39.124355: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:11:39.335205: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_39 2025-05-20 02:11:39.399688: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_39/marlene.trace.json.gz 2025-05-20 02:11:39.470713: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.101 ms 2025-05-20 02:11:39.504489: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_39Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_39/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_39/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_39/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_39/marlene.kernel_stats.pb 27/80 [=========>....................] - ETA: 1:12 - loss: 1.2420 - rpn_class_loss: 0.0158 - rpn_bbox_loss: 0.4951 - mrcnn_class_loss: 0.0285 - mrcnn_bbox_loss: 0.3452 - mrcnn_mask_loss: 0.35732025-05-20 02:11:42.229230: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/80 [=========>....................] - ETA: 1:09 - loss: 1.2220 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4794 - mrcnn_class_loss: 0.0292 - mrcnn_bbox_loss: 0.3376 - mrcnn_mask_loss: 0.35972025-05-20 02:11:42.839678: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:42.840313: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3426 activity events. 2025-05-20 02:11:43.031112: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_42 2025-05-20 02:11:43.145265: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_42/marlene.trace.json.gz 2025-05-20 02:11:43.240329: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.663 ms 2025-05-20 02:11:43.284344: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_42Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_42/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_42/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_42/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_42/marlene.kernel_stats.pb 29/80 [=========>....................] - ETA: 1:16 - loss: 1.2155 - rpn_class_loss: 0.0163 - rpn_bbox_loss: 0.4752 - mrcnn_class_loss: 0.0288 - mrcnn_bbox_loss: 0.3371 - mrcnn_mask_loss: 0.35802025-05-20 02:11:48.533206: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/80 [==========>...................] - ETA: 1:12 - loss: 1.2062 - rpn_class_loss: 0.0166 - rpn_bbox_loss: 0.4685 - mrcnn_class_loss: 0.0285 - mrcnn_bbox_loss: 0.3348 - mrcnn_mask_loss: 0.35782025-05-20 02:11:48.840207: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:48.841345: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:11:49.108471: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_48 2025-05-20 02:11:49.241602: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_48/marlene.trace.json.gz 2025-05-20 02:11:49.384349: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.818 ms 2025-05-20 02:11:49.460020: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_48Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_48/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_48/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_48/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_48/marlene.kernel_stats.pb 31/80 [==========>...................] - ETA: 1:15 - loss: 1.1928 - rpn_class_loss: 0.0164 - rpn_bbox_loss: 0.4628 - mrcnn_class_loss: 0.0280 - mrcnn_bbox_loss: 0.3297 - mrcnn_mask_loss: 0.35592025-05-20 02:11:53.226745: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/80 [===========>..................] - ETA: 1:12 - loss: 1.1783 - rpn_class_loss: 0.0162 - rpn_bbox_loss: 0.4564 - mrcnn_class_loss: 0.0278 - mrcnn_bbox_loss: 0.3238 - mrcnn_mask_loss: 0.35402025-05-20 02:11:53.523018: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:53.523922: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3462 callback api events and 3462 activity events. 2025-05-20 02:11:53.697683: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_53 2025-05-20 02:11:53.757205: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_53/marlene.trace.json.gz 2025-05-20 02:11:53.806704: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.922 ms 2025-05-20 02:11:53.832554: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_53Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_53/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_53/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_53/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_53/marlene.kernel_stats.pb 33/80 [===========>..................] - ETA: 1:10 - loss: 1.1690 - rpn_class_loss: 0.0168 - rpn_bbox_loss: 0.4475 - mrcnn_class_loss: 0.0274 - mrcnn_bbox_loss: 0.3269 - mrcnn_mask_loss: 0.35052025-05-20 02:11:54.802168: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/80 [===========>..................] - ETA: 1:08 - loss: 1.1556 - rpn_class_loss: 0.0170 - rpn_bbox_loss: 0.4366 - mrcnn_class_loss: 0.0273 - mrcnn_bbox_loss: 0.3279 - mrcnn_mask_loss: 0.34682025-05-20 02:11:55.553222: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:55.553945: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3423 callback api events and 3423 activity events. 2025-05-20 02:11:55.704038: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_55 2025-05-20 02:11:55.767980: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_55/marlene.trace.json.gz 2025-05-20 02:11:55.824669: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.27 ms 2025-05-20 02:11:55.852473: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_55Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_55/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_55/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_55/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_55/marlene.kernel_stats.pb 35/80 [============>.................] - ETA: 1:06 - loss: 1.1548 - rpn_class_loss: 0.0170 - rpn_bbox_loss: 0.4363 - mrcnn_class_loss: 0.0269 - mrcnn_bbox_loss: 0.3287 - mrcnn_mask_loss: 0.34582025-05-20 02:11:57.013955: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/80 [============>.................] - ETA: 1:04 - loss: 1.1509 - rpn_class_loss: 0.0170 - rpn_bbox_loss: 0.4355 - mrcnn_class_loss: 0.0267 - mrcnn_bbox_loss: 0.3270 - mrcnn_mask_loss: 0.34482025-05-20 02:11:58.084718: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:11:58.085636: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3388 activity events. 2025-05-20 02:11:58.230749: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_58 2025-05-20 02:11:58.290516: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_58/marlene.trace.json.gz 2025-05-20 02:11:58.353326: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.939 ms 2025-05-20 02:11:58.388500: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_58Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_58/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_58/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_58/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_11_58/marlene.kernel_stats.pb 37/80 [============>.................] - ETA: 1:03 - loss: 1.1601 - rpn_class_loss: 0.0170 - rpn_bbox_loss: 0.4452 - mrcnn_class_loss: 0.0265 - mrcnn_bbox_loss: 0.3250 - mrcnn_mask_loss: 0.34642025-05-20 02:11:59.464557: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/80 [=============>................] - ETA: 1:00 - loss: 1.1567 - rpn_class_loss: 0.0170 - rpn_bbox_loss: 0.4494 - mrcnn_class_loss: 0.0260 - mrcnn_bbox_loss: 0.3182 - mrcnn_mask_loss: 0.34612025-05-20 02:12:00.205951: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:00.206678: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:12:00.341296: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_00 2025-05-20 02:12:00.403377: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_00/marlene.trace.json.gz 2025-05-20 02:12:00.459335: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.94 ms 2025-05-20 02:12:00.502655: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_00Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_00/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_00/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_00/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_00/marlene.kernel_stats.pb 39/80 [=============>................] - ETA: 1:00 - loss: 1.1387 - rpn_class_loss: 0.0171 - rpn_bbox_loss: 0.4387 - mrcnn_class_loss: 0.0256 - mrcnn_bbox_loss: 0.3129 - mrcnn_mask_loss: 0.34442025-05-20 02:12:02.759130: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 40/80 [==============>...............] - ETA: 58s - loss: 1.1212 - rpn_class_loss: 0.0173 - rpn_bbox_loss: 0.4284 - mrcnn_class_loss: 0.0253 - mrcnn_bbox_loss: 0.3074 - mrcnn_mask_loss: 0.3428 2025-05-20 02:12:03.710286: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:03.711907: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3348 callback api events and 3348 activity events. 2025-05-20 02:12:03.855855: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_03 2025-05-20 02:12:03.914975: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_03/marlene.trace.json.gz 2025-05-20 02:12:03.964593: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.92 ms 2025-05-20 02:12:03.991452: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_03Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_03/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_03/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_03/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_03/marlene.kernel_stats.pb 41/80 [==============>...............] - ETA: 56s - loss: 1.0968 - rpn_class_loss: 0.0173 - rpn_bbox_loss: 0.4204 - mrcnn_class_loss: 0.0247 - mrcnn_bbox_loss: 0.2999 - mrcnn_mask_loss: 0.33452025-05-20 02:12:05.122355: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 42/80 [==============>...............] - ETA: 54s - loss: 1.0764 - rpn_class_loss: 0.0176 - rpn_bbox_loss: 0.4154 - mrcnn_class_loss: 0.0241 - mrcnn_bbox_loss: 0.2928 - mrcnn_mask_loss: 0.32652025-05-20 02:12:05.903603: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:05.905024: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3279 callback api events and 3279 activity events. 2025-05-20 02:12:06.103087: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_06 2025-05-20 02:12:06.165097: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_06/marlene.trace.json.gz 2025-05-20 02:12:06.219884: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.871 ms 2025-05-20 02:12:06.256082: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_06Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_06/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_06/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_06/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_06/marlene.kernel_stats.pb 43/80 [===============>..............] - ETA: 53s - loss: 1.0691 - rpn_class_loss: 0.0173 - rpn_bbox_loss: 0.4133 - mrcnn_class_loss: 0.0240 - mrcnn_bbox_loss: 0.2910 - mrcnn_mask_loss: 0.32352025-05-20 02:12:07.756986: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 44/80 [===============>..............] - ETA: 51s - loss: 1.0598 - rpn_class_loss: 0.0169 - rpn_bbox_loss: 0.4104 - mrcnn_class_loss: 0.0238 - mrcnn_bbox_loss: 0.2886 - mrcnn_mask_loss: 0.32012025-05-20 02:12:08.711779: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:08.715320: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3427 activity events. 2025-05-20 02:12:08.936481: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_08 2025-05-20 02:12:09.015325: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_08/marlene.trace.json.gz 2025-05-20 02:12:09.068462: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.07 ms 2025-05-20 02:12:09.109330: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_08Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_08/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_08/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_08/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_08/marlene.kernel_stats.pb 45/80 [===============>..............] - ETA: 51s - loss: 1.0553 - rpn_class_loss: 0.0168 - rpn_bbox_loss: 0.4026 - mrcnn_class_loss: 0.0238 - mrcnn_bbox_loss: 0.2913 - mrcnn_mask_loss: 0.32092025-05-20 02:12:10.861324: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 46/80 [================>.............] - ETA: 49s - loss: 1.0498 - rpn_class_loss: 0.0166 - rpn_bbox_loss: 0.3951 - mrcnn_class_loss: 0.0238 - mrcnn_bbox_loss: 0.2928 - mrcnn_mask_loss: 0.32162025-05-20 02:12:11.760645: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:11.762612: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3388 activity events. 2025-05-20 02:12:12.000080: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_11 2025-05-20 02:12:12.074385: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_11/marlene.trace.json.gz 2025-05-20 02:12:12.139139: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.578 ms 2025-05-20 02:12:12.182507: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_11Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_11/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_11/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_11/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_11/marlene.kernel_stats.pb /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 47/80 [================>.............] - ETA: 50s - loss: 1.0544 - rpn_class_loss: 0.0164 - rpn_bbox_loss: 0.4069 - mrcnn_class_loss: 0.0235 - mrcnn_bbox_loss: 0.2887 - mrcnn_mask_loss: 0.31892025-05-20 02:12:16.580735: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 48/80 [=================>............] - ETA: 48s - loss: 1.0582 - rpn_class_loss: 0.0163 - rpn_bbox_loss: 0.4176 - mrcnn_class_loss: 0.0236 - mrcnn_bbox_loss: 0.2844 - mrcnn_mask_loss: 0.31632025-05-20 02:12:17.921295: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:17.923516: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3387 activity events. 2025-05-20 02:12:18.146589: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_18 2025-05-20 02:12:18.247704: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_18/marlene.trace.json.gz 2025-05-20 02:12:18.331349: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.78 ms 2025-05-20 02:12:18.373695: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_18Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_18/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_18/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_18/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_18/marlene.kernel_stats.pb 49/80 [=================>............] - ETA: 47s - loss: 1.0744 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4254 - mrcnn_class_loss: 0.0257 - mrcnn_bbox_loss: 0.2875 - mrcnn_mask_loss: 0.31962025-05-20 02:12:20.712288: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 50/80 [=================>............] - ETA: 45s - loss: 1.0834 - rpn_class_loss: 0.0159 - rpn_bbox_loss: 0.4243 - mrcnn_class_loss: 0.0267 - mrcnn_bbox_loss: 0.2898 - mrcnn_mask_loss: 0.32672025-05-20 02:12:21.365035: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:21.365896: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3463 activity events. 2025-05-20 02:12:21.533227: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_21 2025-05-20 02:12:21.630493: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_21/marlene.trace.json.gz 2025-05-20 02:12:21.713398: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.58 ms 2025-05-20 02:12:21.758248: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_21Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_21/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_21/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_21/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_21/marlene.kernel_stats.pb 51/80 [==================>...........] - ETA: 45s - loss: 1.0793 - rpn_class_loss: 0.0159 - rpn_bbox_loss: 0.4186 - mrcnn_class_loss: 0.0264 - mrcnn_bbox_loss: 0.2917 - mrcnn_mask_loss: 0.32662025-05-20 02:12:25.318568: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 52/80 [==================>...........] - ETA: 43s - loss: 1.0749 - rpn_class_loss: 0.0160 - rpn_bbox_loss: 0.4131 - mrcnn_class_loss: 0.0262 - mrcnn_bbox_loss: 0.2930 - mrcnn_mask_loss: 0.32662025-05-20 02:12:25.781455: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:25.782671: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3462 callback api events and 3462 activity events. 2025-05-20 02:12:26.154845: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_26 2025-05-20 02:12:26.306385: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_26/marlene.trace.json.gz 2025-05-20 02:12:26.474863: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.579 ms 2025-05-20 02:12:26.606649: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_26/marlene.kernel_stats.pb 53/80 [==================>...........] - ETA: 42s - loss: 1.0608 - rpn_class_loss: 0.0160 - rpn_bbox_loss: 0.4112 - mrcnn_class_loss: 0.0257 - mrcnn_bbox_loss: 0.2875 - mrcnn_mask_loss: 0.32042025-05-20 02:12:29.263866: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 54/80 [===================>..........] - ETA: 40s - loss: 1.0466 - rpn_class_loss: 0.0162 - rpn_bbox_loss: 0.4085 - mrcnn_class_loss: 0.0252 - mrcnn_bbox_loss: 0.2822 - mrcnn_mask_loss: 0.31452025-05-20 02:12:30.136297: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:30.138704: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3279 callback api events and 3281 activity events. 2025-05-20 02:12:30.394109: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_30 2025-05-20 02:12:30.510444: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_30/marlene.trace.json.gz 2025-05-20 02:12:30.672888: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.19 ms 2025-05-20 02:12:30.742503: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_30Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_30/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_30/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_30/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_30/marlene.kernel_stats.pb 55/80 [===================>..........] - ETA: 41s - loss: 1.0486 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4117 - mrcnn_class_loss: 0.0253 - mrcnn_bbox_loss: 0.2826 - mrcnn_mask_loss: 0.31292025-05-20 02:12:35.803077: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 56/80 [====================>.........] - ETA: 38s - loss: 1.0498 - rpn_class_loss: 0.0160 - rpn_bbox_loss: 0.4144 - mrcnn_class_loss: 0.0252 - mrcnn_bbox_loss: 0.2828 - mrcnn_mask_loss: 0.31142025-05-20 02:12:36.166630: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:36.168668: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3424 callback api events and 3424 activity events. 2025-05-20 02:12:36.399662: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_36 2025-05-20 02:12:36.485646: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_36/marlene.trace.json.gz 2025-05-20 02:12:36.547237: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.883 ms 2025-05-20 02:12:36.571541: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_36Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_36/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_36/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_36/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_36/marlene.kernel_stats.pb 57/80 [====================>.........] - ETA: 38s - loss: 1.0485 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4162 - mrcnn_class_loss: 0.0249 - mrcnn_bbox_loss: 0.2819 - mrcnn_mask_loss: 0.30952025-05-20 02:12:40.065465: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 58/80 [====================>.........] - ETA: 36s - loss: 1.0438 - rpn_class_loss: 0.0162 - rpn_bbox_loss: 0.4166 - mrcnn_class_loss: 0.0246 - mrcnn_bbox_loss: 0.2786 - mrcnn_mask_loss: 0.30792025-05-20 02:12:40.729815: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:40.732094: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3388 activity events. 2025-05-20 02:12:40.971004: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_40 2025-05-20 02:12:41.076159: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_40/marlene.trace.json.gz 2025-05-20 02:12:41.167897: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.786 ms 2025-05-20 02:12:41.224736: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_40Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_40/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_40/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_40/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_40/marlene.kernel_stats.pb 59/80 [=====================>........] - ETA: 35s - loss: 1.0434 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4172 - mrcnn_class_loss: 0.0245 - mrcnn_bbox_loss: 0.2788 - mrcnn_mask_loss: 0.30672025-05-20 02:12:44.026598: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 60/80 [=====================>........] - ETA: 33s - loss: 1.0368 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4128 - mrcnn_class_loss: 0.0248 - mrcnn_bbox_loss: 0.2775 - mrcnn_mask_loss: 0.30562025-05-20 02:12:45.319608: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:45.321230: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:12:45.559254: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_45 2025-05-20 02:12:45.629048: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_45/marlene.trace.json.gz 2025-05-20 02:12:45.701811: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.491 ms 2025-05-20 02:12:45.733080: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_45Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_45/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_45/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_45/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_45/marlene.kernel_stats.pb 61/80 [=====================>........] - ETA: 34s - loss: 1.0337 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4091 - mrcnn_class_loss: 0.0253 - mrcnn_bbox_loss: 0.2774 - mrcnn_mask_loss: 0.30572025-05-20 02:12:54.399109: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 62/80 [======================>.......] - ETA: 31s - loss: 1.0285 - rpn_class_loss: 0.0160 - rpn_bbox_loss: 0.4050 - mrcnn_class_loss: 0.0259 - mrcnn_bbox_loss: 0.2762 - mrcnn_mask_loss: 0.30542025-05-20 02:12:54.729022: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:12:54.730845: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3387 activity events. 2025-05-20 02:12:54.995593: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_54 2025-05-20 02:12:55.097008: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_54/marlene.trace.json.gz 2025-05-20 02:12:55.260432: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.635 ms 2025-05-20 02:12:55.299981: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_54Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_54/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_54/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_54/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_12_54/marlene.kernel_stats.pb 63/80 [======================>.......] - ETA: 31s - loss: 1.0232 - rpn_class_loss: 0.0159 - rpn_bbox_loss: 0.4020 - mrcnn_class_loss: 0.0259 - mrcnn_bbox_loss: 0.2741 - mrcnn_mask_loss: 0.30522025-05-20 02:13:00.878618: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 64/80 [=======================>......] - ETA: 28s - loss: 1.0173 - rpn_class_loss: 0.0158 - rpn_bbox_loss: 0.3989 - mrcnn_class_loss: 0.0259 - mrcnn_bbox_loss: 0.2714 - mrcnn_mask_loss: 0.30522025-05-20 02:13:01.262081: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:01.263438: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3386 activity events. 2025-05-20 02:13:01.579640: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_01 2025-05-20 02:13:01.701980: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_01/marlene.trace.json.gz 2025-05-20 02:13:01.804242: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.297 ms 2025-05-20 02:13:01.859805: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_01Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_01/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_01/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_01/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_01/marlene.kernel_stats.pb 65/80 [=======================>......] - ETA: 26s - loss: 1.0126 - rpn_class_loss: 0.0157 - rpn_bbox_loss: 0.3972 - mrcnn_class_loss: 0.0260 - mrcnn_bbox_loss: 0.2695 - mrcnn_mask_loss: 0.30422025-05-20 02:13:02.190258: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 66/80 [=======================>......] - ETA: 26s - loss: 1.0061 - rpn_class_loss: 0.0156 - rpn_bbox_loss: 0.3931 - mrcnn_class_loss: 0.0263 - mrcnn_bbox_loss: 0.2682 - mrcnn_mask_loss: 0.30282025-05-20 02:13:08.904680: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:08.906017: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3433 activity events. 2025-05-20 02:13:09.246962: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_09 2025-05-20 02:13:09.468583: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_09/marlene.trace.json.gz 2025-05-20 02:13:09.733496: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.938 ms 2025-05-20 02:13:09.837734: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_09/marlene.kernel_stats.pb 67/80 [========================>.....] - ETA: 24s - loss: 1.0165 - rpn_class_loss: 0.0154 - rpn_bbox_loss: 0.3943 - mrcnn_class_loss: 0.0272 - mrcnn_bbox_loss: 0.2746 - mrcnn_mask_loss: 0.30482025-05-20 02:13:10.327858: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 68/80 [========================>.....] - ETA: 22s - loss: 1.0300 - rpn_class_loss: 0.0152 - rpn_bbox_loss: 0.3946 - mrcnn_class_loss: 0.0285 - mrcnn_bbox_loss: 0.2820 - mrcnn_mask_loss: 0.30972025-05-20 02:13:13.240861: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:13.241672: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3467 activity events. 2025-05-20 02:13:13.467596: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_13 2025-05-20 02:13:13.553208: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_13/marlene.trace.json.gz 2025-05-20 02:13:13.649471: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.12 ms 2025-05-20 02:13:13.683726: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_13Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_13/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_13/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_13/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_13/marlene.kernel_stats.pb 69/80 [========================>.....] - ETA: 20s - loss: 1.0366 - rpn_class_loss: 0.0150 - rpn_bbox_loss: 0.3948 - mrcnn_class_loss: 0.0292 - mrcnn_bbox_loss: 0.2860 - mrcnn_mask_loss: 0.31172025-05-20 02:13:14.470651: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 70/80 [=========================>....] - ETA: 18s - loss: 1.0323 - rpn_class_loss: 0.0150 - rpn_bbox_loss: 0.3914 - mrcnn_class_loss: 0.0292 - mrcnn_bbox_loss: 0.2862 - mrcnn_mask_loss: 0.31052025-05-20 02:13:16.856479: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:16.859247: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3389 activity events. 2025-05-20 02:13:17.140345: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_16 2025-05-20 02:13:17.254384: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_16/marlene.trace.json.gz 2025-05-20 02:13:17.363260: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.879 ms 2025-05-20 02:13:17.413266: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_16/marlene.kernel_stats.pb 71/80 [=========================>....] - ETA: 16s - loss: 1.0406 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.3914 - mrcnn_class_loss: 0.0301 - mrcnn_bbox_loss: 0.2889 - mrcnn_mask_loss: 0.31532025-05-20 02:13:19.086213: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 72/80 [==========================>...] - ETA: 15s - loss: 1.0886 - rpn_class_loss: 0.0149 - rpn_bbox_loss: 0.4340 - mrcnn_class_loss: 0.0304 - mrcnn_bbox_loss: 0.2967 - mrcnn_mask_loss: 0.31272025-05-20 02:13:22.004743: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:22.008540: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3467 activity events. 2025-05-20 02:13:22.474532: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_22 2025-05-20 02:13:22.708351: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_22/marlene.trace.json.gz 2025-05-20 02:13:22.800088: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.956 ms 2025-05-20 02:13:22.854362: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_22Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_22/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_22/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_22/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_22/marlene.kernel_stats.pb 73/80 [==========================>...] - ETA: 13s - loss: 1.0828 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.4289 - mrcnn_class_loss: 0.0307 - mrcnn_bbox_loss: 0.2962 - mrcnn_mask_loss: 0.31222025-05-20 02:13:28.133839: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 74/80 [==========================>...] - ETA: 11s - loss: 1.0957 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.4267 - mrcnn_class_loss: 0.0319 - mrcnn_bbox_loss: 0.3086 - mrcnn_mask_loss: 0.31372025-05-20 02:13:28.535647: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:28.538578: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3425 callback api events and 3425 activity events. 2025-05-20 02:13:28.801052: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_28 2025-05-20 02:13:28.901970: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_28/marlene.trace.json.gz 2025-05-20 02:13:29.096835: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.388 ms 2025-05-20 02:13:29.156178: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_28Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_28/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_28/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_28/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_28/marlene.kernel_stats.pb 75/80 [===========================>..] - ETA: 9s - loss: 1.0929 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.4258 - mrcnn_class_loss: 0.0316 - mrcnn_bbox_loss: 0.3075 - mrcnn_mask_loss: 0.3131 2025-05-20 02:13:33.775675: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 76/80 [===========================>..] - ETA: 7s - loss: 1.1317 - rpn_class_loss: 0.0149 - rpn_bbox_loss: 0.4615 - mrcnn_class_loss: 0.0318 - mrcnn_bbox_loss: 0.3127 - mrcnn_mask_loss: 0.31092025-05-20 02:13:34.186695: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:34.192528: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3463 callback api events and 3463 activity events. 2025-05-20 02:13:34.899888: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_34 2025-05-20 02:13:35.127589: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_34/marlene.trace.json.gz 2025-05-20 02:13:35.287728: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.434 ms 2025-05-20 02:13:35.421871: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_34/marlene.kernel_stats.pb 77/80 [===========================>..] - ETA: 5s - loss: 1.1428 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.4593 - mrcnn_class_loss: 0.0327 - mrcnn_bbox_loss: 0.3221 - mrcnn_mask_loss: 0.31392025-05-20 02:13:38.407401: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 78/80 [============================>.] - ETA: 3s - loss: 1.1407 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.4599 - mrcnn_class_loss: 0.0325 - mrcnn_bbox_loss: 0.3207 - mrcnn_mask_loss: 0.31292025-05-20 02:13:41.049658: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:41.052031: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3386 callback api events and 3390 activity events. 2025-05-20 02:13:41.443485: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_41 2025-05-20 02:13:41.590612: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_41/marlene.trace.json.gz 2025-05-20 02:13:41.773478: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.328 ms 2025-05-20 02:13:41.832197: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_41Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_41/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_41/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_41/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_41/marlene.kernel_stats.pb 79/80 [============================>.] - ETA: 2s - loss: 1.1399 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.4570 - mrcnn_class_loss: 0.0325 - mrcnn_bbox_loss: 0.3229 - mrcnn_mask_loss: 0.31282025-05-20 02:13:43.853128: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:13:45.734185: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:13:45.735135: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3387 callback api events and 3390 activity events. 2025-05-20 02:13:45.975855: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_45 2025-05-20 02:13:46.080258: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_45/marlene.trace.json.gz 2025-05-20 02:13:46.178793: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.464 ms 2025-05-20 02:13:46.225752: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_45Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_45/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_45/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_45/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_13_45/marlene.kernel_stats.pb 80/80 [==============================] - 162s 2s/step - loss: 1.1389 - rpn_class_loss: 0.0147 - rpn_bbox_loss: 0.4598 - mrcnn_class_loss: 0.0323 - mrcnn_bbox_loss: 0.3205 - mrcnn_mask_loss: 0.3115 - val_loss: 1.3093 - val_rpn_class_loss: 0.0094 - val_rpn_bbox_loss: 0.8474 - val_mrcnn_class_loss: 0.0239 - val_mrcnn_bbox_loss: 0.2136 - val_mrcnn_mask_loss: 0.2150 2025-05-20 02:13:48.957884: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:13:49.014828: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed Training all layers Starting at epoch 2. LR=0.0005 Checkpoint Path: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/mask_rcnn_learn_mask_pancarte_200525_1_{epoch:04d}.h5 Selecting layers to train conv1 (Conv2D) bn_conv1 (BatchNorm) res2a_branch2a (Conv2D) bn2a_branch2a (BatchNorm) res2a_branch2b (Conv2D) bn2a_branch2b (BatchNorm) res2a_branch2c (Conv2D) res2a_branch1 (Conv2D) bn2a_branch2c (BatchNorm) bn2a_branch1 (BatchNorm) res2b_branch2a (Conv2D) bn2b_branch2a (BatchNorm) res2b_branch2b (Conv2D) bn2b_branch2b (BatchNorm) res2b_branch2c (Conv2D) bn2b_branch2c (BatchNorm) res2c_branch2a (Conv2D) bn2c_branch2a (BatchNorm) res2c_branch2b (Conv2D) bn2c_branch2b (BatchNorm) res2c_branch2c (Conv2D) bn2c_branch2c (BatchNorm) res3a_branch2a (Conv2D) bn3a_branch2a (BatchNorm) res3a_branch2b (Conv2D) bn3a_branch2b (BatchNorm) res3a_branch2c (Conv2D) res3a_branch1 (Conv2D) bn3a_branch2c (BatchNorm) bn3a_branch1 (BatchNorm) res3b_branch2a (Conv2D) bn3b_branch2a (BatchNorm) res3b_branch2b (Conv2D) bn3b_branch2b (BatchNorm) res3b_branch2c (Conv2D) bn3b_branch2c (BatchNorm) res3c_branch2a (Conv2D) bn3c_branch2a (BatchNorm) res3c_branch2b (Conv2D) bn3c_branch2b (BatchNorm) res3c_branch2c (Conv2D) bn3c_branch2c (BatchNorm) res3d_branch2a (Conv2D) bn3d_branch2a (BatchNorm) res3d_branch2b (Conv2D) bn3d_branch2b (BatchNorm) res3d_branch2c (Conv2D) bn3d_branch2c (BatchNorm) res4a_branch2a (Conv2D) bn4a_branch2a (BatchNorm) res4a_branch2b (Conv2D) bn4a_branch2b (BatchNorm) res4a_branch2c (Conv2D) res4a_branch1 (Conv2D) bn4a_branch2c (BatchNorm) bn4a_branch1 (BatchNorm) res4b_branch2a (Conv2D) bn4b_branch2a (BatchNorm) res4b_branch2b (Conv2D) bn4b_branch2b (BatchNorm) res4b_branch2c (Conv2D) bn4b_branch2c (BatchNorm) res4c_branch2a (Conv2D) bn4c_branch2a (BatchNorm) res4c_branch2b (Conv2D) bn4c_branch2b (BatchNorm) res4c_branch2c (Conv2D) bn4c_branch2c (BatchNorm) res4d_branch2a (Conv2D) bn4d_branch2a (BatchNorm) res4d_branch2b (Conv2D) bn4d_branch2b (BatchNorm) res4d_branch2c (Conv2D) bn4d_branch2c (BatchNorm) res4e_branch2a (Conv2D) bn4e_branch2a (BatchNorm) res4e_branch2b (Conv2D) bn4e_branch2b (BatchNorm) res4e_branch2c (Conv2D) bn4e_branch2c (BatchNorm) res4f_branch2a (Conv2D) bn4f_branch2a (BatchNorm) res4f_branch2b (Conv2D) bn4f_branch2b (BatchNorm) res4f_branch2c (Conv2D) bn4f_branch2c (BatchNorm) res4g_branch2a (Conv2D) bn4g_branch2a (BatchNorm) res4g_branch2b (Conv2D) bn4g_branch2b (BatchNorm) res4g_branch2c (Conv2D) bn4g_branch2c (BatchNorm) res4h_branch2a (Conv2D) bn4h_branch2a (BatchNorm) res4h_branch2b (Conv2D) bn4h_branch2b (BatchNorm) res4h_branch2c (Conv2D) bn4h_branch2c (BatchNorm) res4i_branch2a (Conv2D) bn4i_branch2a (BatchNorm) res4i_branch2b (Conv2D) bn4i_branch2b (BatchNorm) res4i_branch2c (Conv2D) bn4i_branch2c (BatchNorm) res4j_branch2a (Conv2D) bn4j_branch2a (BatchNorm) res4j_branch2b (Conv2D) bn4j_branch2b (BatchNorm) res4j_branch2c (Conv2D) bn4j_branch2c (BatchNorm) res4k_branch2a (Conv2D) bn4k_branch2a (BatchNorm) res4k_branch2b (Conv2D) bn4k_branch2b (BatchNorm) res4k_branch2c (Conv2D) bn4k_branch2c (BatchNorm) res4l_branch2a (Conv2D) bn4l_branch2a (BatchNorm) res4l_branch2b (Conv2D) bn4l_branch2b (BatchNorm) res4l_branch2c (Conv2D) bn4l_branch2c (BatchNorm) res4m_branch2a (Conv2D) bn4m_branch2a (BatchNorm) res4m_branch2b (Conv2D) bn4m_branch2b (BatchNorm) res4m_branch2c (Conv2D) bn4m_branch2c (BatchNorm) res4n_branch2a (Conv2D) bn4n_branch2a (BatchNorm) res4n_branch2b (Conv2D) bn4n_branch2b (BatchNorm) res4n_branch2c (Conv2D) bn4n_branch2c (BatchNorm) res4o_branch2a (Conv2D) bn4o_branch2a (BatchNorm) res4o_branch2b (Conv2D) bn4o_branch2b (BatchNorm) res4o_branch2c (Conv2D) bn4o_branch2c (BatchNorm) res4p_branch2a (Conv2D) bn4p_branch2a (BatchNorm) res4p_branch2b (Conv2D) bn4p_branch2b (BatchNorm) res4p_branch2c (Conv2D) bn4p_branch2c (BatchNorm) res4q_branch2a (Conv2D) bn4q_branch2a (BatchNorm) res4q_branch2b (Conv2D) bn4q_branch2b (BatchNorm) res4q_branch2c (Conv2D) bn4q_branch2c (BatchNorm) res4r_branch2a (Conv2D) bn4r_branch2a (BatchNorm) res4r_branch2b (Conv2D) bn4r_branch2b (BatchNorm) res4r_branch2c (Conv2D) bn4r_branch2c (BatchNorm) res4s_branch2a (Conv2D) bn4s_branch2a (BatchNorm) res4s_branch2b (Conv2D) bn4s_branch2b (BatchNorm) res4s_branch2c (Conv2D) bn4s_branch2c (BatchNorm) res4t_branch2a (Conv2D) bn4t_branch2a (BatchNorm) res4t_branch2b (Conv2D) bn4t_branch2b (BatchNorm) res4t_branch2c (Conv2D) bn4t_branch2c (BatchNorm) res4u_branch2a (Conv2D) bn4u_branch2a (BatchNorm) res4u_branch2b (Conv2D) bn4u_branch2b (BatchNorm) res4u_branch2c (Conv2D) bn4u_branch2c (BatchNorm) res4v_branch2a (Conv2D) bn4v_branch2a (BatchNorm) res4v_branch2b (Conv2D) bn4v_branch2b (BatchNorm) res4v_branch2c (Conv2D) bn4v_branch2c (BatchNorm) res4w_branch2a (Conv2D) bn4w_branch2a (BatchNorm) res4w_branch2b (Conv2D) bn4w_branch2b (BatchNorm) res4w_branch2c (Conv2D) bn4w_branch2c (BatchNorm) res5a_branch2a (Conv2D) bn5a_branch2a (BatchNorm) res5a_branch2b (Conv2D) bn5a_branch2b (BatchNorm) res5a_branch2c (Conv2D) res5a_branch1 (Conv2D) bn5a_branch2c (BatchNorm) bn5a_branch1 (BatchNorm) res5b_branch2a (Conv2D) bn5b_branch2a (BatchNorm) res5b_branch2b (Conv2D) bn5b_branch2b (BatchNorm) res5b_branch2c (Conv2D) bn5b_branch2c (BatchNorm) res5c_branch2a (Conv2D) bn5c_branch2a (BatchNorm) res5c_branch2b (Conv2D) bn5c_branch2b (BatchNorm) res5c_branch2c (Conv2D) bn5c_branch2c (BatchNorm) fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.275476). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.235910). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.233214). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.284223). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.278650). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.275855). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.358377). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.435856). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.665383). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.672895). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.575052). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.419451). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.415807). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.415296). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.415756). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.419120). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.483815). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.240594). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.239802). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.204730). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.202466). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.230298). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.231494). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.254984). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.255733). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.265521). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.284182). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.336220). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.296981). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.283085). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.278419). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.338900). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.315946). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/engine/training_generator.py:48: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class. warnings.warn( WARNING:tensorflow:Model failed to serialize as JSON. Ignoring... cannot pickle '_thread.RLock' object mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Epoch 3/6 1/80 [..............................] - ETA: 3:52:50 - loss: 0.5576 - rpn_class_loss: 0.0215 - rpn_bbox_loss: 0.2900 - mrcnn_class_loss: 0.0072 - mrcnn_bbox_loss: 0.1540 - mrcnn_mask_loss: 0.08492025-05-20 02:20:01.807881: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/80 [..............................] - ETA: 1:56:01 - loss: 0.4942 - rpn_class_loss: 0.0210 - rpn_bbox_loss: 0.2440 - mrcnn_class_loss: 0.0057 - mrcnn_bbox_loss: 0.1173 - mrcnn_mask_loss: 0.10622025-05-20 02:20:03.580747: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:03.612984: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9962 callback api events and 9965 activity events. 2025-05-20 02:20:05.345696: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_04 2025-05-20 02:20:05.960792: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_04/marlene.trace.json.gz 2025-05-20 02:20:06.645158: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.982 ms 2025-05-20 02:20:07.104752: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_04Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_04/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_04/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_04/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_04/marlene.kernel_stats.pb 3/80 [>.............................] - ETA: 1:18:28 - loss: 0.9112 - rpn_class_loss: 0.0144 - rpn_bbox_loss: 0.3491 - mrcnn_class_loss: 0.0141 - mrcnn_bbox_loss: 0.2719 - mrcnn_mask_loss: 0.26172025-05-20 02:20:08.335673: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 4/80 [>.............................] - ETA: 58:29 - loss: 1.7653 - rpn_class_loss: 0.0109 - rpn_bbox_loss: 0.9871 - mrcnn_class_loss: 0.0171 - mrcnn_bbox_loss: 0.3804 - mrcnn_mask_loss: 0.3698 2025-05-20 02:20:09.707924: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:09.714922: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10039 callback api events and 10042 activity events. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 2025-05-20 02:20:10.508290: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_10 2025-05-20 02:20:10.827818: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_10/marlene.trace.json.gz 2025-05-20 02:20:11.235550: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.632 ms 2025-05-20 02:20:11.470268: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_10Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_10/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_10/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_10/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_10/marlene.kernel_stats.pb 5/80 [>.............................] - ETA: 46:52 - loss: 1.5708 - rpn_class_loss: 0.0104 - rpn_bbox_loss: 0.8453 - mrcnn_class_loss: 0.0203 - mrcnn_bbox_loss: 0.3508 - mrcnn_mask_loss: 0.34392025-05-20 02:20:12.388595: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/80 [=>............................] - ETA: 38:50 - loss: 1.4186 - rpn_class_loss: 0.0100 - rpn_bbox_loss: 0.7535 - mrcnn_class_loss: 0.0191 - mrcnn_bbox_loss: 0.3145 - mrcnn_mask_loss: 0.32152025-05-20 02:20:14.009023: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:14.024644: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10039 callback api events and 10039 activity events. 2025-05-20 02:20:15.587016: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_15 2025-05-20 02:20:15.878042: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_15/marlene.trace.json.gz 2025-05-20 02:20:16.478716: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.691 ms 2025-05-20 02:20:16.768445: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_15Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_15/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_15/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_15/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_15/marlene.kernel_stats.pb 7/80 [=>............................] - ETA: 33:30 - loss: 1.2906 - rpn_class_loss: 0.0093 - rpn_bbox_loss: 0.6549 - mrcnn_class_loss: 0.0206 - mrcnn_bbox_loss: 0.3058 - mrcnn_mask_loss: 0.29992025-05-20 02:20:17.687268: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/80 [==>...........................] - ETA: 29:05 - loss: 1.1828 - rpn_class_loss: 0.0087 - rpn_bbox_loss: 0.5778 - mrcnn_class_loss: 0.0231 - mrcnn_bbox_loss: 0.2911 - mrcnn_mask_loss: 0.28212025-05-20 02:20:19.018977: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:19.036183: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10039 callback api events and 10039 activity events. 2025-05-20 02:20:20.465956: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_19 2025-05-20 02:20:20.847491: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_19/marlene.trace.json.gz 2025-05-20 02:20:21.308286: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.837 ms 2025-05-20 02:20:21.601268: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_19Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_19/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_19/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_19/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_19/marlene.kernel_stats.pb 9/80 [==>...........................] - ETA: 25:59 - loss: 1.1106 - rpn_class_loss: 0.0093 - rpn_bbox_loss: 0.5341 - mrcnn_class_loss: 0.0240 - mrcnn_bbox_loss: 0.2723 - mrcnn_mask_loss: 0.27082025-05-20 02:20:22.581666: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/80 [==>...........................] - ETA: 23:12 - loss: 1.0453 - rpn_class_loss: 0.0095 - rpn_bbox_loss: 0.4980 - mrcnn_class_loss: 0.0243 - mrcnn_bbox_loss: 0.2505 - mrcnn_mask_loss: 0.26292025-05-20 02:20:24.072352: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:24.096063: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10001 callback api events and 10004 activity events. 2025-05-20 02:20:25.253328: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_24 2025-05-20 02:20:25.495167: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_24/marlene.trace.json.gz 2025-05-20 02:20:26.002615: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.858 ms 2025-05-20 02:20:26.485369: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_24Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_24/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_24/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_24/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_24/marlene.kernel_stats.pb 11/80 [===>..........................] - ETA: 21:13 - loss: 1.0053 - rpn_class_loss: 0.0112 - rpn_bbox_loss: 0.4811 - mrcnn_class_loss: 0.0226 - mrcnn_bbox_loss: 0.2396 - mrcnn_mask_loss: 0.25082025-05-20 02:20:27.888962: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/80 [===>..........................] - ETA: 19:17 - loss: 0.9597 - rpn_class_loss: 0.0122 - rpn_bbox_loss: 0.4418 - mrcnn_class_loss: 0.0213 - mrcnn_bbox_loss: 0.2324 - mrcnn_mask_loss: 0.25202025-05-20 02:20:29.264755: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:29.271846: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9963 callback api events and 9966 activity events. 2025-05-20 02:20:31.462375: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_30 2025-05-20 02:20:32.335683: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_30/marlene.trace.json.gz 2025-05-20 02:20:33.005709: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.853 ms 2025-05-20 02:20:33.488767: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_30Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_30/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_30/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_30/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_30/marlene.kernel_stats.pb 13/80 [===>..........................] - ETA: 18:01 - loss: 0.9253 - rpn_class_loss: 0.0134 - rpn_bbox_loss: 0.4258 - mrcnn_class_loss: 0.0201 - mrcnn_bbox_loss: 0.2233 - mrcnn_mask_loss: 0.24272025-05-20 02:20:34.686613: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/80 [====>.........................] - ETA: 16:34 - loss: 0.8976 - rpn_class_loss: 0.0136 - rpn_bbox_loss: 0.3970 - mrcnn_class_loss: 0.0211 - mrcnn_bbox_loss: 0.2218 - mrcnn_mask_loss: 0.24412025-05-20 02:20:36.338261: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:36.362904: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10001 callback api events and 10005 activity events. 2025-05-20 02:20:37.688482: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_37 2025-05-20 02:20:37.973010: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_37/marlene.trace.json.gz 2025-05-20 02:20:38.514368: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.981 ms 2025-05-20 02:20:38.769914: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_37Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_37/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_37/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_37/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_37/marlene.kernel_stats.pb 15/80 [====>.........................] - ETA: 15:30 - loss: 0.8769 - rpn_class_loss: 0.0140 - rpn_bbox_loss: 0.3771 - mrcnn_class_loss: 0.0218 - mrcnn_bbox_loss: 0.2196 - mrcnn_mask_loss: 0.24442025-05-20 02:20:39.664856: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/80 [=====>........................] - ETA: 14:23 - loss: 0.8566 - rpn_class_loss: 0.0141 - rpn_bbox_loss: 0.3600 - mrcnn_class_loss: 0.0208 - mrcnn_bbox_loss: 0.2203 - mrcnn_mask_loss: 0.24132025-05-20 02:20:41.030104: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:41.051264: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9961 callback api events and 9961 activity events. 2025-05-20 02:20:42.914398: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_42 2025-05-20 02:20:43.630622: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_42/marlene.trace.json.gz 2025-05-20 02:20:44.366060: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.031 ms 2025-05-20 02:20:44.799125: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_42Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_42/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_42/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_42/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_42/marlene.kernel_stats.pb 17/80 [=====>........................] - ETA: 13:39 - loss: 0.8539 - rpn_class_loss: 0.0137 - rpn_bbox_loss: 0.3687 - mrcnn_class_loss: 0.0206 - mrcnn_bbox_loss: 0.2138 - mrcnn_mask_loss: 0.23712025-05-20 02:20:46.021863: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/80 [=====>........................] - ETA: 12:48 - loss: 0.8468 - rpn_class_loss: 0.0133 - rpn_bbox_loss: 0.3738 - mrcnn_class_loss: 0.0199 - mrcnn_bbox_loss: 0.2065 - mrcnn_mask_loss: 0.23332025-05-20 02:20:48.609522: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:48.624829: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10004 callback api events and 10008 activity events. 2025-05-20 02:20:51.127211: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_50 2025-05-20 02:20:51.411990: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_50/marlene.trace.json.gz 2025-05-20 02:20:51.833816: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.857 ms 2025-05-20 02:20:52.140917: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_50Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_50/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_50/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_50/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_50/marlene.kernel_stats.pb 19/80 [======>.......................] - ETA: 12:13 - loss: 0.8279 - rpn_class_loss: 0.0127 - rpn_bbox_loss: 0.3601 - mrcnn_class_loss: 0.0208 - mrcnn_bbox_loss: 0.2036 - mrcnn_mask_loss: 0.23052025-05-20 02:20:53.400489: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/80 [======>.......................] - ETA: 11:32 - loss: 0.8100 - rpn_class_loss: 0.0123 - rpn_bbox_loss: 0.3509 - mrcnn_class_loss: 0.0209 - mrcnn_bbox_loss: 0.1977 - mrcnn_mask_loss: 0.22822025-05-20 02:20:56.011195: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:20:56.033081: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10001 callback api events and 10005 activity events. 2025-05-20 02:20:58.207181: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_57 2025-05-20 02:20:58.822632: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_57/marlene.trace.json.gz 2025-05-20 02:21:00.247286: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 18.839 ms 2025-05-20 02:21:00.625931: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_57Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_57/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_57/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_57/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_20_57/marlene.kernel_stats.pb 21/80 [======>.......................] - ETA: 11:08 - loss: 0.8259 - rpn_class_loss: 0.0125 - rpn_bbox_loss: 0.3531 - mrcnn_class_loss: 0.0294 - mrcnn_bbox_loss: 0.1977 - mrcnn_mask_loss: 0.23322025-05-20 02:21:02.785376: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/80 [=======>......................] - ETA: 10:32 - loss: 0.8400 - rpn_class_loss: 0.0126 - rpn_bbox_loss: 0.3546 - mrcnn_class_loss: 0.0360 - mrcnn_bbox_loss: 0.1994 - mrcnn_mask_loss: 0.23732025-05-20 02:21:05.529475: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:21:05.552927: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10042 callback api events and 10046 activity events. 2025-05-20 02:21:08.110554: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_07 2025-05-20 02:21:08.474146: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_07/marlene.trace.json.gz 2025-05-20 02:21:09.192967: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 20.408 ms 2025-05-20 02:21:09.704398: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_07Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_07/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_07/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_07/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_07/marlene.kernel_stats.pb 23/80 [=======>......................] - ETA: 10:09 - loss: 0.8307 - rpn_class_loss: 0.0125 - rpn_bbox_loss: 0.3527 - mrcnn_class_loss: 0.0348 - mrcnn_bbox_loss: 0.1956 - mrcnn_mask_loss: 0.23512025-05-20 02:21:10.854322: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/80 [========>.....................] - ETA: 9:36 - loss: 0.8240 - rpn_class_loss: 0.0125 - rpn_bbox_loss: 0.3518 - mrcnn_class_loss: 0.0337 - mrcnn_bbox_loss: 0.1930 - mrcnn_mask_loss: 0.2328 2025-05-20 02:21:12.388891: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:21:12.405935: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10000 callback api events and 10004 activity events. 2025-05-20 02:21:14.214818: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_13 2025-05-20 02:21:14.925744: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_13/marlene.trace.json.gz 2025-05-20 02:21:15.360930: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.146 ms 2025-05-20 02:21:15.605203: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_13Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_13/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_13/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_13/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_13/marlene.kernel_stats.pb 25/80 [========>.....................] - ETA: 9:13 - loss: 0.8108 - rpn_class_loss: 0.0126 - rpn_bbox_loss: 0.3431 - mrcnn_class_loss: 0.0327 - mrcnn_bbox_loss: 0.1911 - mrcnn_mask_loss: 0.23132025-05-20 02:21:16.560958: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/80 [========>.....................] - ETA: 8:45 - loss: 0.7970 - rpn_class_loss: 0.0127 - rpn_bbox_loss: 0.3336 - mrcnn_class_loss: 0.0318 - mrcnn_bbox_loss: 0.1879 - mrcnn_mask_loss: 0.23092025-05-20 02:21:18.512319: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:21:18.526064: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10000 callback api events and 10004 activity events. 2025-05-20 02:21:20.847422: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_19 2025-05-20 02:21:21.420456: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_19/marlene.trace.json.gz 2025-05-20 02:21:22.311848: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 27.048 ms 2025-05-20 02:21:23.429474: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_19Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_19/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_19/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_19/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_19/marlene.kernel_stats.pb 27/80 [=========>....................] - ETA: 8:30 - loss: 0.7830 - rpn_class_loss: 0.0125 - rpn_bbox_loss: 0.3236 - mrcnn_class_loss: 0.0314 - mrcnn_bbox_loss: 0.1856 - mrcnn_mask_loss: 0.22992025-05-20 02:21:24.710346: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/80 [=========>....................] - ETA: 8:04 - loss: 0.7761 - rpn_class_loss: 0.0123 - rpn_bbox_loss: 0.3204 - mrcnn_class_loss: 0.0310 - mrcnn_bbox_loss: 0.1852 - mrcnn_mask_loss: 0.22712025-05-20 02:21:26.289624: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:21:26.299576: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9963 callback api events and 9968 activity events. 2025-05-20 02:21:27.612159: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_26 2025-05-20 02:21:28.025997: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_26/marlene.trace.json.gz 2025-05-20 02:21:28.972843: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 13.133 ms 2025-05-20 02:21:29.415756: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_26/marlene.kernel_stats.pb 29/80 [=========>....................] - ETA: 7:46 - loss: 0.7647 - rpn_class_loss: 0.0121 - rpn_bbox_loss: 0.3122 - mrcnn_class_loss: 0.0306 - mrcnn_bbox_loss: 0.1823 - mrcnn_mask_loss: 0.22752025-05-20 02:21:30.152098: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/80 [==========>...................] - ETA: 7:25 - loss: 0.7584 - rpn_class_loss: 0.0121 - rpn_bbox_loss: 0.3039 - mrcnn_class_loss: 0.0306 - mrcnn_bbox_loss: 0.1858 - mrcnn_mask_loss: 0.22592025-05-20 02:21:32.668367: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:21:32.689267: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10039 callback api events and 10044 activity events. 2025-05-20 02:21:35.888115: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_34 2025-05-20 02:21:36.504375: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_34/marlene.trace.json.gz 2025-05-20 02:21:37.411099: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 11.152 ms 2025-05-20 02:21:37.806565: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_34/marlene.kernel_stats.pb 31/80 [==========>...................] - ETA: 7:12 - loss: 0.7479 - rpn_class_loss: 0.0119 - rpn_bbox_loss: 0.2972 - mrcnn_class_loss: 0.0301 - mrcnn_bbox_loss: 0.1854 - mrcnn_mask_loss: 0.22332025-05-20 02:21:38.799889: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/80 [===========>..................] - ETA: 6:52 - loss: 0.7396 - rpn_class_loss: 0.0117 - rpn_bbox_loss: 0.2920 - mrcnn_class_loss: 0.0295 - mrcnn_bbox_loss: 0.1849 - mrcnn_mask_loss: 0.22142025-05-20 02:21:40.134439: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:21:40.144080: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10001 callback api events and 10006 activity events. 2025-05-20 02:21:42.006245: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_41 2025-05-20 02:21:42.464157: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_41/marlene.trace.json.gz 2025-05-20 02:21:43.418746: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 35.028 ms 2025-05-20 02:21:44.025729: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_41Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_41/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_41/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_41/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_41/marlene.kernel_stats.pb 33/80 [===========>..................] - ETA: 6:39 - loss: 0.7323 - rpn_class_loss: 0.0115 - rpn_bbox_loss: 0.2895 - mrcnn_class_loss: 0.0294 - mrcnn_bbox_loss: 0.1824 - mrcnn_mask_loss: 0.21952025-05-20 02:21:45.387177: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/80 [===========>..................] - ETA: 6:21 - loss: 0.7279 - rpn_class_loss: 0.0113 - rpn_bbox_loss: 0.2884 - mrcnn_class_loss: 0.0289 - mrcnn_bbox_loss: 0.1811 - mrcnn_mask_loss: 0.21822025-05-20 02:21:47.033411: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:21:47.056903: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10001 callback api events and 10006 activity events. 2025-05-20 02:21:50.231485: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_49 2025-05-20 02:21:50.987595: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_49/marlene.trace.json.gz 2025-05-20 02:21:51.510428: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.868 ms 2025-05-20 02:21:52.083715: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_49Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_49/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_49/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_49/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_49/marlene.kernel_stats.pb 35/80 [============>.................] - ETA: 6:10 - loss: 0.7212 - rpn_class_loss: 0.0112 - rpn_bbox_loss: 0.2857 - mrcnn_class_loss: 0.0291 - mrcnn_bbox_loss: 0.1795 - mrcnn_mask_loss: 0.21572025-05-20 02:21:53.310042: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/80 [============>.................] - ETA: 5:53 - loss: 0.7127 - rpn_class_loss: 0.0109 - rpn_bbox_loss: 0.2823 - mrcnn_class_loss: 0.0294 - mrcnn_bbox_loss: 0.1771 - mrcnn_mask_loss: 0.21292025-05-20 02:21:54.687555: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:21:54.726232: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10001 callback api events and 10006 activity events. 2025-05-20 02:21:57.360469: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_56 2025-05-20 02:21:58.007574: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_56/marlene.trace.json.gz 2025-05-20 02:21:58.740832: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 14.708 ms 2025-05-20 02:21:59.156026: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_56Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_56/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_56/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_56/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_21_56/marlene.kernel_stats.pb 37/80 [============>.................] - ETA: 5:43 - loss: 0.7169 - rpn_class_loss: 0.0111 - rpn_bbox_loss: 0.2793 - mrcnn_class_loss: 0.0295 - mrcnn_bbox_loss: 0.1809 - mrcnn_mask_loss: 0.21602025-05-20 02:22:00.087108: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/80 [=============>................] - ETA: 5:26 - loss: 0.7112 - rpn_class_loss: 0.0114 - rpn_bbox_loss: 0.2737 - mrcnn_class_loss: 0.0292 - mrcnn_bbox_loss: 0.1805 - mrcnn_mask_loss: 0.21652025-05-20 02:22:00.874630: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:22:00.885320: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9963 callback api events and 9963 activity events. 2025-05-20 02:22:02.433638: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_01 2025-05-20 02:22:03.017584: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_01/marlene.trace.json.gz 2025-05-20 02:22:04.019493: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.579 ms 2025-05-20 02:22:04.483634: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_01Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_01/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_01/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_01/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_01/marlene.kernel_stats.pb 39/80 [=============>................] - ETA: 5:16 - loss: 0.7091 - rpn_class_loss: 0.0115 - rpn_bbox_loss: 0.2747 - mrcnn_class_loss: 0.0287 - mrcnn_bbox_loss: 0.1799 - mrcnn_mask_loss: 0.21432025-05-20 02:22:05.793071: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 40/80 [==============>...............] - ETA: 5:02 - loss: 0.7058 - rpn_class_loss: 0.0116 - rpn_bbox_loss: 0.2743 - mrcnn_class_loss: 0.0285 - mrcnn_bbox_loss: 0.1794 - mrcnn_mask_loss: 0.21202025-05-20 02:22:08.518473: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:22:08.543412: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10001 callback api events and 10001 activity events. 2025-05-20 02:22:10.737872: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_09 2025-05-20 02:22:11.324961: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_09/marlene.trace.json.gz 2025-05-20 02:22:12.285012: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 9.362 ms 2025-05-20 02:22:12.765408: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_1/learn_mask_pancarte_200525_120250520T0159/train/plugins/profile/2025_05_20_02_22_09/marlene.kernel_stats.pb 41/80 [==============>...............] - ETA: 4:53 - loss: 0.7129 - rpn_class_loss: 0.0115 - rpn_bbox_loss: 0.2838 - mrcnn_class_loss: 0.0285 - mrcnn_bbox_loss: 0.1784 - mrcnn_mask_loss: 0.21072025-05-20 02:22:13.925798: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 42/80 [==============>...............] - ETA: 4:40 - loss: 0.7165 - rpn_class_loss: 0.0113 - rpn_bbox_loss: 0.2911 - mrcnn_class_loss: 0.0283 - mrcnn_bbox_loss: 0.1765 - mrcnn_mask_loss: 0.2094Command terminated by signal 11 986.23user 116.40system 24:08.00elapsed 76%CPU (0avgtext+0avgdata 3688064maxresident)k 2727968inputs+2664072outputs (92203major+10607691minor)pagefaults 0swaps /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Process ForkPoolWorker-3: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 404, in _send_bytes self._send(header) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Process ForkPoolWorker-4: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 404, in _send_bytes self._send(header) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Process ForkPoolWorker-1: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 405, in _send_bytes self._send(buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Process ForkPoolWorker-2: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 404, in _send_bytes self._send(header) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe