python /home/admin/mtr/script_for_cron.py -j default -m 10 -a 'python3 ~/workarea/git/Velours/python/mtr/mask_rcnn/mask_rcnn_train.py --name=learn_mask_pancarte_200525_2 --hashtag_type=5007 -P 23147753 --class_name=badge,pancarte --steps_per_epoch=40 --lr=0.0005 ' -s learn_mask -M 31 -S 0 -U 100,100,120 2025-05-20 02:24:34.224108: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-20 02:24:34.259069: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-20 02:24:34.260868: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f499c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-20 02:24:34.260932: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-20 02:24:34.265859: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-20 02:24:34.538866: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xc61ed00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-20 02:24:34.538972: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-20 02:24:34.540260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-20 02:24:34.541356: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-20 02:24:34.563708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-20 02:24:34.580314: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-20 02:24:34.584160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-20 02:24:34.606883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-20 02:24:34.610793: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-20 02:24:34.638281: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-20 02:24:34.640577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-20 02:24:34.640720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-20 02:24:34.643867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-20 02:24:34.643969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-20 02:24:34.644008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-20 02:24:34.648540: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_rcnn_train.py:33: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. model_param file didn't exist model_name : mask_coco_origin model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : [] file manque in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] local folder : /data/models_weight/mask_coco_origin import MySQLdb succeeded Import error (python version) warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.0005 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_mask_pancarte_200525_2 NUM_CLASSES 3 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 40 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 100 WEIGHT_DECAY 0.0001 launch train use defaut coco pre_trained model : /data/models_weight/mask_coco_origin/mask_model.h5 492993645 496005930 batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +{492993645: 'badge', 496005930: 'pancarte'} badge 492993645 1356331534 x0 : 639 y1 : 2294 width : 2076, height : 1311, area : 2721636, score : 1.0 None pancarte 496005930 1356331534 x0 : -1 y1 : 4158 width : 3125, height : 2735, area : 8546875, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331534.jpg pancarte 496005930 1356331538 x0 : 1319 y1 : 2643 width : 1206, height : 826, area : 996156, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331538.jpg pancarte 496005930 1356331545 x0 : 1109 y1 : 1843 width : 546, height : 431, area : 235326, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331545.jpg pancarte 496005930 1356331553 x0 : 971 y1 : 1646 width : 888, height : 679, area : 602952, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331553.jpg pancarte 496005930 1356331557 x0 : 1707 y1 : 1808 width : 774, height : 617, area : 477558, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331557.jpg pancarte 496005930 1356331559 x0 : 2057 y1 : 1619 width : 670, height : 514, area : 344380, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331559.jpg pancarte 496005930 1356331574 x0 : 0 y1 : 4159 width : 3119, height : 3332, area : 10392508, score : 1.0 None badge 492993645 1356331574 x0 : 772 y1 : 2309 width : 2341, height : 1582, area : 3703462, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331574.jpg pancarte 496005930 1356331578 x0 : 1710 y1 : 2479 width : 1107, height : 869, area : 961983, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331578.jpg pancarte 496005930 1356331586 x0 : 768 y1 : 2082 width : 840, height : 639, area : 536760, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331586.jpg pancarte 496005930 1356331593 x0 : 1150 y1 : 2168 width : 994, height : 818, area : 813092, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331593.jpg pancarte 496005930 1356331598 x0 : 1303 y1 : 2819 width : 1093, height : 982, area : 1073326, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331598.jpg pancarte 496005930 1356331602 x0 : 1052 y1 : 2716 width : 1124, height : 996, area : 1119504, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331602.jpg pancarte 496005930 1356331619 x0 : 2024 y1 : 2197 width : 725, height : 604, area : 437900, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331619.jpg badge 492993645 1356331626 x0 : 2158 y1 : 1396 width : 661, height : 455, area : 300755, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331626.jpg badge 492993645 1356331643 x0 : 1799 y1 : 1208 width : 976, height : 652, area : 636352, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331643.jpg pancarte 496005930 1356331654 x0 : 1386 y1 : 2579 width : 963, height : 832, area : 801216, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331654.jpg pancarte 496005930 1356331661 x0 : 1304 y1 : 2433 width : 896, height : 665, area : 595840, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331661.jpg badge 492993645 1356331664 x0 : 390 y1 : 3823 width : 2211, height : 3267, area : 7223337, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331664.jpg pancarte 496005930 1356331673 x0 : 1495 y1 : 1563 width : 1086, height : 723, area : 785178, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331673.jpg pancarte 496005930 1356331680 x0 : 2210 y1 : 1252 width : 302, height : 241, area : 72782, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331680.jpg pancarte 496005930 1356331687 x0 : 2909 y1 : 1573 width : 847, height : 607, area : 514129, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331687.jpg pancarte 496005930 1356331694 x0 : 2557 y1 : 1303 width : 360, height : 309, area : 111240, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331694.jpg pancarte 496005930 1356331701 x0 : 1174 y1 :pancarte 496005930 1356331931 x0 : 1854 y1 : 1823 width : 484, height : 392, area : 189728, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331931.jpg pancarte 496005930 1356331943 x0 : 1826 y1 : 1571 width : 738, height : 572, area : 422136, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331943.jpg pancarte 496005930 1356331946 x0 : 2088 y1 : 1812 width : 782, height : 635, area : 496570, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331946.jpg pancarte 496005930 1356331957 x0 : 1804 y1 : 2287 width : 1318, height : 770, area : 1014860, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331957.jpg badge 492993645 1356331962 x0 : 9 y1 : 2615 width : 4151, height : 2608, area : 10825808, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331962.jpg pancarte 496005930 1356331967 x0 : 1452 y1 : 2869 width : 1218, height : 1081, area : 1316658, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331967.jpg pancarte 496005930 1356331976 x0 : 1331 y1 : 1960 width : 676, height : 511, area : 345436, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331976.jpg pancarte 496005930 1356331982 x0 : 1110 y1 : 2192 width : 693, height : 589, area : 408177, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331982.jpg pancarte 496005930 1356331985 x0 : 1999 y1 : 1817 width : 604, height : 480, area : 289920, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331985.jpg pancarte 496005930 1356331995 x0 : 1953 y1 : 2553 width : 1224, height : 817, area : 1000008, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356331995.jpg badge 492993645 1356332003 x0 : 2 y1 : 2685 width : 4149, height : 2685, area : 11140065, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356332003.jpg pancarte 496005930 1356332010 x0 : 2111 y1 : 2391 width : 861, height : 606, area : 521766, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356332010.jpg pancarte 496005930 1356332017 x0 : 2273 y1 : 2433 width : 939, height : 678, area : 636642, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356332017.jpg pancarte 496005930 1356332023 x0 : 1082 y1 : 1448 width : 366, height : 283, area : 103578, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356332023.jpg pancarte 496005930 1356332025 x0 : 2597 y1 : 2307 width : 504, height : 1056, area : 532224, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356332025.jpg pancarte 496005930 1356332042 x0 : 1872 y1 : 2500 width : 771, height : 549, area : 423279, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356332042.jpg pancarte 496005930 1356361517 x0 : 2274 y1 : 2363 width : 1381, height : 872, area : 1204232, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356361517.jpg badge 492993645 1356361523 x0 : 2010 y1 : 1624 width : 712, height : 445, area : 316840, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356361523.jpg pancarte 496005930 1356361530 x0 : 1595 y1 : 2256 width : 729, height : 597, area : 435213, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356361530.jpg pancarte 496005930 1356361537 x0 : 1900 y1 : 2125 width : 903, height : 724, area : 653772, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356361537.jpg pancarte 496005930 1356361543 x0 : 2208 y1 : 2389 width : 1015, height : 858, area : 870870, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356361543.jpg pancarte 496005930 1356361548 x0 : 1707 y1 : 2804 width : 1011, height : 880, area : 889680, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356361548.jpg pancarte 496005930 1356982967 x0 : 1773 y1 : 2444 width : 1347, height : 815, area : 1097805, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356982967.jpg badge 492993645 1356982971 x0 : 710 y1 : 4147 width : 2395, height : 3474, area : 8320230, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356982971.jpg pancarte 496005930 1356982975 x0 : 2092 y1 : 2226 width : 630, height : 502, area : 316260, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356982975.jpg pancarte 496005930 1356982983 x0 : 2134 y1 : 2335 width : 850, height : 675, area : 573750, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356982983.jpg pancarte 496005930 1356982991 x0 : 2351 y1 : 2008 width : 659, height : 532, area : 350588, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356982991.jpg pancarte 496005930 1356982997 x0 : 1309 y1 : 2176 width : 682, height : 510, area : 347820, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356982997.jpg pancarte 496005930 1356983040 x0 : 2101 y1 : 2457 width : 1047, height : 834, area : 873198, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356983040.jpg pancarte 496005930 1356983046 x0 : 1961 y1 : 2890 width : 1298, height : 1056, area : 1370688, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356983046.jpg badge 492993645 1356983052 x0 : 697 y1 : 2314 width : 2560, height : 1594, area : 4080640, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356983052.jpg pancarte 496005930 1356983061 x0 : 1541 y1 : 2587 width : 717, height : 552, area : 395784, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356983061.jpg pancarte 496005930 1356983067 x0 : 1264 y1 : 2104 width : 720, height : 543, area : 390960, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356983067.jpg pancarte 496005930 1356983071 x0 : 1181 y1 : 1546 width : 441, height : 298, area : 131418, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356983071.jpg pancarte 496005930 1356983086 x0 : 1603 y1 : 2607 width : 739, height : 540, area : 399060, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356983086.jpg pancarte 496005930 1356998184 x0 : 2033 y1 : 2402 width : 1383, height : 930, area : 1286190, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998184.jpg pancarte 496005930 1356998222 x0 : 2246 y1 : 2293 width : 635, height : 531, area : 337185, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998222.jpg pancarte 496005930 1356998225 x0 : 1619 y1 : 3994 width : 1188, height : 897, area : 1065636, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998225.jpg pancarte 496005930 1356998228 x0 : 1872 y1 : 2231 width : 671, height : 483, area : 324093, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998228.jpg pancarte 496005930 1356998234 x0 : 1252 y1 : 2346 width : 537, height : 290, area : 155730, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998234.jpg pancarte 496005930 1356998239 x0 : 1994 y1 : 1998 width : 515, height : 395, area : 203425, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998239.jpg pancarte 496005930 1356998244 x0 : 932 y1 : 2481 width : 665, height : 519, area : 345135, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998244.jpg pancarte 496005930 1356998320 x0 : 1577 y1 : 1513 width : 269, height : 241, area : 64829, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998320.jpg pancarte 496005930 1356998437 x0 : 1634 y1 : 3112 width : 950, height : 852, area : 809400, score : 1.0 None pancarte 496005930 1356998437 x0 : 1633 y1 : 3108 width : 945, height : 849, area : 802305, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998437.jpg pancarte 496005930 1356998545 x0 : 1214 y1 : 1860 width : 401, height : 363, area : 145563, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998545.jpg pancarte 496005930 1356998665 x0 : 1165 y1 : 2263 width : 904, height : 650, area : 587600, score : 1.0 None Using TensorFlow backend. WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1659: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. 2025-05-20 02:26:10.781677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-20 02:26:10.781820: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-20 02:26:10.781844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-20 02:26:10.781864: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-20 02:26:10.781883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-20 02:26:10.781899: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-20 02:26:10.781914: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-20 02:26:10.781931: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-20 02:26:10.783095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:570: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead 2025-05-20 02:26:17.635912: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:26:17.636133: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1363] Profiler found 1 GPUs 2025-05-20 02:26:17.643339: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcupti.so.10.1 2025-05-20 02:26:17.714555: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:26:31.462148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-20 02:26:31.462336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-20 02:26:31.462366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-20 02:26:31.462399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-20 02:26:31.462428: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-20 02:26:31.462455: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-20 02:26:31.462478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-20 02:26:31.462507: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-20 02:26:31.463833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-20 02:26:31.463921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-20 02:26:31.463937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-20 02:26:31.463951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-20 02:26:31.466010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/engine/training_generator.py:48: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class. warnings.warn( WARNING:tensorflow:Model failed to serialize as JSON. Ignoring... cannot pickle '_thread.RLock' object /data/data_root/learn_mask_pancarte_200525_2/train/1356998665.jpg badge 492993645 1356998732 x0 : 409 y1 : 4159 width : 2665, height : 4065, area : 10833225, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/train/1356998732.jpg pancarte 496005930 1356998753 x0 : 2109 y1 : 2432 width : 1295, height : 864, area : 1118880, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1356998753.jpg pancarte 496005930 1356998806 x0 : 1661 y1 : 1549 width : 608, height : 523, area : 317984, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1356998806.jpg pancarte 496005930 1356998813 x0 : 899 y1 : 1307 width : 1136, height : 1121, area : 1273456, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1356998813.jpg pancarte 496005930 1356998821 x0 : 1208 y1 : 1234 width : 482, height : 388, area : 187016, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1356998821.jpg pancarte 496005930 1356998826 x0 : 1050 y1 : 1365 width : 389, height : 322, area : 125258, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1356998826.jpg pancarte 496005930 1356998832 x0 : 1729 y1 : 1676 width : 470, height : 411, area : 193170, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1356998832.jpg pancarte 496005930 1357676318 x0 : 1942 y1 : 2041 width : 1334, height : 816, area : 1088544, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1357676318.jpg pancarte 496005930 1357676324 x0 : 1992 y1 : 2636 width : 928, height : 643, area : 596704, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1357676324.jpg pancarte 496005930 1357676331 x0 : 1758 y1 : 2800 width : 1140, height : 797, area : 908580, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1357676331.jpg pancarte 496005930 1357676341 x0 : 1756 y1 : 2940 width : 888, height : 948, area : 841824, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1357676341.jpg pancarte 496005930 1357676347 x0 : 1714 y1 : 3069 width : 1170, height : 1037, area : 1213290, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1357676347.jpg pancarte 496005930 1357676358 x0 : 1940 y1 : 2721 width : 882, height : 676, area : 596232, score : 1.0 None /data/data_root/learn_mask_pancarte_200525_2/val/1357676358.jpg number of images :103 number of images :12 Training network heads Starting at epoch 0. LR=0.0005 Checkpoint Path: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/mask_rcnn_learn_mask_pancarte_200525_2_{epoch:04d}.h5 Selecting layers to train fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) 2025-05-20 02:27:02.825844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-20 02:27:03.433871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 Epoch 1/2 1/40 [..............................] - ETA: 11:35 - loss: 4.7580 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 1.1111 - mrcnn_class_loss: 0.5080 - mrcnn_bbox_loss: 1.3333 - mrcnn_mask_loss: 1.79782025-05-20 02:27:09.371901: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 5:46 - loss: 4.6251 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 1.1029 - mrcnn_class_loss: 0.4613 - mrcnn_bbox_loss: 1.2995 - mrcnn_mask_loss: 1.7537 2025-05-20 02:27:09.765995: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:27:09.768541: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3442 callback api events and 3442 activity events. 2025-05-20 02:27:10.000577: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_09 2025-05-20 02:27:10.079561: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_09/marlene.trace.json.gz 2025-05-20 02:27:10.133360: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.883 ms 2025-05-20 02:27:10.158630: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_09/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 3:52 - loss: 3.7036 - rpn_class_loss: 0.0729 - rpn_bbox_loss: 0.7695 - mrcnn_class_loss: 0.3669 - mrcnn_bbox_loss: 1.1665 - mrcnn_mask_loss: 1.32792025-05-20 02:27:10.371273: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 2:51 - loss: 3.1921 - rpn_class_loss: 0.1049 - rpn_bbox_loss: 0.6023 - mrcnn_class_loss: 0.3020 - mrcnn_bbox_loss: 1.0674 - mrcnn_mask_loss: 1.11552025-05-20 02:27:10.610130: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:27:10.611102: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3327 callback api events and 3327 activity events. 2025-05-20 02:27:10.776794: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_10 2025-05-20 02:27:10.855988: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_10/marlene.trace.json.gz 2025-05-20 02:27:10.926310: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.55 ms 2025-05-20 02:27:10.961919: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_10Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_10/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_10/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_10/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_10/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 2:17 - loss: 2.7865 - rpn_class_loss: 0.0977 - rpn_bbox_loss: 0.6931 - mrcnn_class_loss: 0.2493 - mrcnn_bbox_loss: 0.8539 - mrcnn_mask_loss: 0.89242025-05-20 02:27:11.184236: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 1:52 - loss: 2.5033 - rpn_class_loss: 0.0927 - rpn_bbox_loss: 0.7448 - mrcnn_class_loss: 0.2104 - mrcnn_bbox_loss: 0.7116 - mrcnn_mask_loss: 0.74372025-05-20 02:27:11.444009: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:27:11.445477: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3258 callback api events and 3258 activity events. 2025-05-20 02:27:11.660794: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_11 2025-05-20 02:27:11.751090: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_11/marlene.trace.json.gz 2025-05-20 02:27:11.815270: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.418 ms 2025-05-20 02:27:11.848979: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_11Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_11/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_11/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_11/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_11/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 1:36 - loss: 2.5080 - rpn_class_loss: 0.0809 - rpn_bbox_loss: 0.6990 - mrcnn_class_loss: 0.2201 - mrcnn_bbox_loss: 0.8111 - mrcnn_mask_loss: 0.69682025-05-20 02:27:12.100551: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 1:23 - loss: 2.5352 - rpn_class_loss: 0.0721 - rpn_bbox_loss: 0.6818 - mrcnn_class_loss: 0.2227 - mrcnn_bbox_loss: 0.8967 - mrcnn_mask_loss: 0.66182025-05-20 02:27:12.325196: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:27:12.326096: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3403 callback api events and 3403 activity events. 2025-05-20 02:27:12.513690: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_12 2025-05-20 02:27:12.612868: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_12/marlene.trace.json.gz 2025-05-20 02:27:12.685372: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.503 ms 2025-05-20 02:27:12.721518: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2 10/40 [======>.......................] - ETA: 1:20 - loss: 2.4834 - rpn_class_loss: 0.0728 - rpn_bbox_loss: 0.7043 - mrcnn_class_loss: 0.1824 - mrcnn_bbox_loss: 0.9198 - mrcnn_mask_loss: 0.60412025-05-20 02:27:18.454336: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:27:18.455085: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3365 callback api events and 3365 activity events. 2025-05-20 02:27:18.593631: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_18 2025-05-20 02:27:18.656098: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_18/marlene.trace.json.gz 2025-05-20 02:27:18.721005: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.653 ms 2025-05-20 02:27:18.755387: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_18Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_18/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_18/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_18/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_18/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 1:25 - loss: 2.3750 - rpn_class_loss: 0.0738 - rpn_bbox_loss: 0.7501 - mrcnn_class_loss: 0.1658 - mrcnn_bbox_loss: 0.8362 - mrcnn_mask_loss: 0.54922025-05-20 02:27:23.806064: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 1:19 - loss: 2.3040 - rpn_class_loss: 0.0749 - rpn_bbox_loss: 0.8071 - mrcnn_class_loss: 0.1520 - mrcnn_bbox_loss: 0.7665 - mrcnn_mask_loss: 0.50342025-05-20 02:27:25.417107: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:27:25.418222: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3258 callback api events and 3258 activity events. 2025-05-20 02:27:25.630487: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_25 2025-05-20 02:27:25.726246: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_25/marlene.trace.json.gz 2025-05-20 02:27:25.807131: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.665 ms 2025-05-20 02:27:25.854498: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_25Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_25/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_25/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_25/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_25/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 1:28 - loss: 2.3814 - rpn_class_loss: 0.0702 - rpn_bbox_loss: 0.8778 - mrcnn_class_loss: 0.1657 - mrcnn_bbox_loss: 0.7663 - mrcnn_mask_loss: 0.50142025-05-20 02:27:34.332851: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 1:19 - loss: 2.4124 - rpn_class_loss: 0.0658 - rpn_bbox_loss: 0.9155 - mrcnn_class_loss: 0.1762 - mrcnn_bbox_loss: 0.7538 - mrcnn_mask_loss: 0.50112025-05-20 02:27:34.569082: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:27:34.569959: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3441 callback api events and 3441 activity events. 2025-05-20 02:27:34.776932: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_34 2025-05-20 02:27:34.871505: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_34/marlene.trace.json.gz 2025-05-20 02:27:34.940740: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.727 ms 2025-05-20 02:27:34.978002: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_27_34/marlene.kernel_stats.pb 15/40 [==========>..................$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC$mmoLD;%g?wŷovH0a5*ؒl͛SiyrO7%L]%hk>v1HBd\(e oIx>36BS%( f$h eԎH`ݶ f{FoY@00uMbz-XI$&gf7Ӵu|'K.oP PF.o9B<~.[<٭${1A.bKxL'u8n5e,]HVWw$Cel|zysKi-qݬbk,wnG;~ er͒~' 1`V⦫-*[LK'2@仪n2NƶGi/U'E@`H;J +Jn#6ڴĹGNG'Z!WiNJ@AZ|[$q}iҷQbtTEC directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_04 2025-05-20 02:28:04.774161: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_04/marlene.trace.json.gz 2025-05-20 02:28:04.846376: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.616 ms 2025-05-20 02:28:04.882531: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_04Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_04/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_04/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_04/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_04/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 56s - loss: 2.3269 - rpn_class_loss: 0.0652 - rpn_bbox_loss: 0.8385 - mrcnn_class_loss: 0.1274 - mrcnn_bbox_loss: 0.7878 - mrcnn_mask_loss: 0.50792025-05-20 02:28:08.063883: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 51s - loss: 2.3068 - rpn_class_loss: 0.0636 - rpn_bbox_loss: 0.8410 - mrcnn_class_loss: 0.1234 - mrcnn_bbox_loss: 0.7790 - mrcnn_mask_loss: 0.49992025-05-20 02:28:09.356760: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:09.357426: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3327 callback api events and 3327 activity events. 2025-05-20 02:28:09.480468: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_09 2025-05-20 02:28:09.562842: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_09/marlene.trace.json.gz 2025-05-20 02:28:09.603076: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.153 ms 2025-05-20 02:28:09.639598: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_09/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 49s - loss: 2.2705 - rpn_class_loss: 0.0615 - rpn_bbox_loss: 0.8151 - mrcnn_class_loss: 0.1263 - mrcnn_bbox_loss: 0.7737 - mrcnn_mask_loss: 0.49402025-05-20 02:28:14.343900: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 44s - loss: 2.2369 - rpn_class_loss: 0.0595 - rpn_bbox_loss: 0.7909 - mrcnn_class_loss: 0.1294 - mrcnn_bbox_loss: 0.7685 - mrcnn_mask_loss: 0.48872025-05-20 02:28:14.618473: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:14.619046: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3403 callback api events and 3403 activity events. 2025-05-20 02:28:14.847125: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_14 2025-05-20 02:28:14.962983: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_14/marlene.trace.json.gz 2025-05-20 02:28:15.048971: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.933 ms 2025-05-20 02:28:15.087908: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_14Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_14/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_14/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_14/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_14/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 41s - loss: 2.2522 - rpn_class_loss: 0.0589 - rpn_bbox_loss: 0.7969 - mrcnn_class_loss: 0.1251 - mrcnn_bbox_loss: 0.7784 - mrcnn_mask_loss: 0.49292025-05-20 02:28:17.896976: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 37s - loss: 2.2601 - rpn_class_loss: 0.0584 - rpn_bbox_loss: 0.8020 - mrcnn_class_loss: 0.1211 - mrcnn_bbox_loss: 0.7825 - mrcnn_mask_loss: 0.49612025-05-20 02:28:19.994552: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:19.996052: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3365 callback api events and 3365 activity events. 2025-05-20 02:28:20.166227: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_20 2025-05-20 02:28:20.222059: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_20/marlene.trace.json.gz 2025-05-20 02:28:20.270627: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.224 ms 2025-05-20 02:28:20.300058: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_20Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_20/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_20/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_20/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_20/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 34s - loss: 2.2287 - rpn_class_loss: 0.0571 - rpn_bbox_loss: 0.7850 - mrcnn_class_loss: 0.1220 - mrcnn_bbox_loss: 0.7692 - mrcnn_mask_loss: 0.49542025-05-20 02:28:23.451727: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 30s - loss: 2.1962 - rpn_class_loss: 0.0559 - rpn_bbox_loss: 0.7691 - mrcnn_class_loss: 0.1221 - mrcnn_bbox_loss: 0.7560 - mrcnn_mask_loss: 0.49312025-05-20 02:28:24.499125: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:24.499949: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3403 callback api events and 3403 activity events. 2025-05-20 02:28:24.699373: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_24 2025-05-20 02:28:24.777473: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_24/marlene.trace.json.gz 2025-05-20 02:28:24.842637: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.333 ms 2025-05-20 02:28:24.879791: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_24Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_24/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_24/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_24/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_24/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 28s - loss: 2.2615 - rpn_class_loss: 0.0541 - rpn_bbox_loss: 0.7990 - mrcnn_class_loss: 0.1246 - mrcnn_bbox_loss: 0.7699 - mrcnn_mask_loss: 0.51402025-05-20 02:28:27.975080: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 24s - loss: 2.3157 - rpn_class_loss: 0.0524 - rpn_bbox_loss: 0.8264 - mrcnn_class_loss: 0.1266 - mrcnn_bbox_loss: 0.7799 - mrcnn_mask_loss: 0.53042025-05-20 02:28:30.319482: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:30.320552: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3442 callback api events and 3442 activity events. 2025-05-20 02:28:30.494734: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_30 2025-05-20 02:28:30.564995: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_30/marlene.trace.json.gz 2025-05-20 02:28:30.614143: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.01 ms 2025-05-20 02:28:30.638375: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_30Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_30/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_30/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_30/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_30/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 21s - loss: 2.2984 - rpn_class_loss: 0.0512 - rpn_bbox_loss: 0.8175 - mrcnn_class_loss: 0.1251 - mrcnn_bbox_loss: 0.7806 - mrcnn_mask_loss: 0.52412025-05-20 02:28:32.050944: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 18s - loss: 2.2721 - rpn_class_loss: 0.0501 - rpn_bbox_loss: 0.8008 - mrcnn_class_loss: 0.1257 - mrcnn_bbox_loss: 0.7778 - mrcnn_mask_loss: 0.51782025-05-20 02:28:34.951286: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:34.952020: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3442 callback api events and 3442 activity events. 2025-05-20 02:28:35.077451: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_35 2025-05-20 02:28:35.157383: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_35/marlene.trace.json.gz 2025-05-20 02:28:35.224526: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.201 ms 2025-05-20 02:28:35.255856: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_35Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_35/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_35/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_35/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_35/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 14s - loss: 2.2294 - rpn_class_loss: 0.0492 - rpn_bbox_loss: 0.7996 - mrcnn_class_loss: 0.1221 - mrcnn_bbox_loss: 0.7556 - mrcnn_mask_loss: 0.50302025-05-20 02:28:35.524529: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 11s - loss: 2.1875 - rpn_class_loss: 0.0484 - rpn_bbox_loss: 0.7968 - mrcnn_class_loss: 0.1187 - mrcnn_bbox_loss: 0.7346 - mrcnn_mask_loss: 0.48902025-05-20 02:28:38.872592: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:38.873450: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3258 callback api events and 3258 activity events. 2025-05-20 02:28:38.998159: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_38 2025-05-20 02:28:39.053520: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_38/marlene.trace.json.gz 2025-05-20 02:28:39.116694: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.241 ms 2025-05-20 02:28:39.149549: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_38Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_38/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_38/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_38/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_38/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 8s - loss: 2.1807 - rpn_class_loss: 0.0480 - rpn_bbox_loss: 0.7893 - mrcnn_class_loss: 0.1174 - mrcnn_bbox_loss: 0.7370 - mrcnn_mask_loss: 0.4890 2025-05-20 02:28:41.258644: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 5s - loss: 2.1719 - rpn_class_loss: 0.0477 - rpn_bbox_loss: 0.7819 - mrcnn_class_loss: 0.1163 - mrcnn_bbox_loss: 0.7372 - mrcnn_mask_loss: 0.48892025-05-20 02:28:42.748043: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:42.749348: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3403 callback api events and 3403 activity events. 2025-05-20 02:28:42.929933: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_42 2025-05-20 02:28:43.013304: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_42/marlene.trace.json.gz 2025-05-20 02:28:43.074488: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.319 ms 2025-05-20 02:28:43.102207: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_42Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_42/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_42/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_42/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_42/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 2s - loss: 2.1781 - rpn_class_loss: 0.0469 - rpn_bbox_loss: 0.7852 - mrcnn_class_loss: 0.1190 - mrcnn_bbox_loss: 0.7364 - mrcnn_mask_loss: 0.49062025-05-20 02:28:44.604936: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:28:45.982819: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:28:45.983875: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3445 callback api events and 3445 activity events. 2025-05-20 02:28:46.118467: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_46 2025-05-20 02:28:46.177631: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_46/marlene.trace.json.gz 2025-05-20 02:28:46.224372: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.057 ms 2025-05-20 02:28:46.247194: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_46Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_46/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_46/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_46/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_28_46/marlene.kernel_stats.pb 40/40 [==============================] - 118s 3s/step - loss: 2.1814 - rpn_class_loss: 0.0462 - rpn_bbox_loss: 0.7869 - mrcnn_class_loss: 0.1223 - mrcnn_bbox_loss: 0.7341 - mrcnn_mask_loss: 0.4918 - val_loss: 0.9709 - val_rpn_class_loss: 0.0122 - val_rpn_bbox_loss: 0.1141 - val_mrcnn_class_loss: 0.0887 - val_mrcnn_bbox_loss: 0.4567 - val_mrcnn_mask_loss: 0.2992 /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Epoch 2/2 1/40 [..............................] - ETA: 10s - loss: 1.9411 - rpn_class_loss: 0.0413 - rpn_bbox_loss: 0.9458 - mrcnn_class_loss: 0.0723 - mrcnn_bbox_loss: 0.5772 - mrcnn_mask_loss: 0.30452025-05-20 02:30:04.281510: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 10s - loss: 1.9702 - rpn_class_loss: 0.0268 - rpn_bbox_loss: 0.9990 - mrcnn_class_loss: 0.0561 - mrcnn_bbox_loss: 0.5549 - mrcnn_mask_loss: 0.33332025-05-20 02:30:04.560689: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:04.563812: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3366 callback api events and 3366 activity events. 2025-05-20 02:30:04.791791: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_04 2025-05-20 02:30:04.869799: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_04/marlene.trace.json.gz 2025-05-20 02:30:04.983505: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.033 ms 2025-05-20 02:30:05.049080: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_04Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_04/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_04/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_04/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_04/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 16s - loss: 1.9315 - rpn_class_loss: 0.0234 - rpn_bbox_loss: 0.8083 - mrcnn_class_loss: 0.0572 - mrcnn_bbox_loss: 0.6198 - mrcnn_mask_loss: 0.42272025-05-20 02:30:05.309074: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 13s - loss: 1.8757 - rpn_class_loss: 0.0217 - rpn_bbox_loss: 0.7095 - mrcnn_class_loss: 0.0547 - mrcnn_bbox_loss: 0.6237 - mrcnn_mask_loss: 0.46612025-05-20 02:30:05.557227: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:05.559820: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3364 callback api events and 3364 activity events. 2025-05-20 02:30:05.764960: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_05 2025-05-20 02:30:05.869546: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_05/marlene.trace.json.gz 2025-05-20 02:30:05.956782: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.527 ms 2025-05-20 02:30:06.002166: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_05Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_05/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_05/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_05/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_05/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 15s - loss: 1.6883 - rpn_class_loss: 0.0230 - rpn_bbox_loss: 0.7497 - mrcnn_class_loss: 0.0438 - mrcnn_bbox_loss: 0.4990 - mrcnn_mask_loss: 0.37292025-05-20 02:30:06.238382: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 14s - loss: 1.5837 - rpn_class_loss: 0.0234 - rpn_bbox_loss: 0.7972 - mrcnn_class_loss: 0.0365 - mrcnn_bbox_loss: 0.4158 - mrcnn_mask_loss: 0.31072025-05-20 02:30:06.492963: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:06.495202: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3258 callback api events and 3258 activity events. 2025-05-20 02:30:06.720755: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_06 2025-05-20 02:30:06.826734: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_06/marlene.trace.json.gz 2025-05-20 02:30:06.927524: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.523 ms 2025-05-20 02:30:06.977753: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_06Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_06/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_06/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_06/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_06/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 15s - loss: 1.4859 - rpn_class_loss: 0.0218 - rpn_bbox_loss: 0.7012 - mrcnn_class_loss: 0.0365 - mrcnn_bbox_loss: 0.4085 - mrcnn_mask_loss: 0.31802025-05-20 02:30:07.220638: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 13s - loss: 1.4067 - rpn_class_loss: 0.0206 - rpn_bbox_loss: 0.6285 - mrcnn_class_loss: 0.0388 - mrcnn_bbox_loss: 0.3960 - mrcnn_mask_loss: 0.32272025-05-20 02:30:07.494392: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:07.496459: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3366 callback api events and 3366 activity events. 2025-05-20 02:30:07.676992: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_07 2025-05-20 02:30:07.744104: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_07/marlene.trace.json.gz 2025-05-20 02:30:07.807247: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.152 ms 2025-05-20 02:30:07.843640: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_07Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_07/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_07/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_07/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_07/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 14s - loss: 1.4066 - rpn_class_loss: 0.0233 - rpn_bbox_loss: 0.6110 - mrcnn_class_loss: 0.0382 - mrcnn_bbox_loss: 0.4076 - mrcnn_mask_loss: 0.32642025-05-20 02:30:08.132531: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 13s - loss: 1.3959 - rpn_class_loss: 0.0254 - rpn_bbox_loss: 0.5963 - mrcnn_class_loss: 0.0370 - mrcnn_bbox_loss: 0.4082 - mrcnn_mask_loss: 0.32902025-05-20 02:30:08.393108: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:08.394047: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3364 callback api events and 3364 activity events. 2025-05-20 02:30:08.591075: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_08 2025-05-20 02:30:08.695108: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_08/marlene.trace.json.gz 2025-05-20 02:30:08.796816: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.675 ms 2025-05-20 02:30:08.844520: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_08Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_08/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_08/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_08/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_08/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 13s - loss: 1.3630 - rpn_class_loss: 0.0247 - rpn_bbox_loss: 0.5709 - mrcnn_class_loss: 0.0373 - mrcnn_bbox_loss: 0.4031 - mrcnn_mask_loss: 0.32702025-05-20 02:30:09.091006: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 12s - loss: 1.3238 - rpn_class_loss: 0.0240 - rpn_bbox_loss: 0.5482 - mrcnn_class_loss: 0.0388 - mrcnn_bbox_loss: 0.3876 - mrcnn_mask_loss: 0.32512025-05-20 02:30:09.359756: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:09.360824: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3403 callback api events and 3403 activity events. 2025-05-20 02:30:09.567347: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_09 2025-05-20 02:30:09.668592: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_09/marlene.trace.json.gz 2025-05-20 02:30:09.756761: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.696 ms 2025-05-20 02:30:09.804149: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_09/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 12s - loss: 1.3143 - rpn_class_loss: 0.0239 - rpn_bbox_loss: 0.5353 - mrcnn_class_loss: 0.0381 - mrcnn_bbox_loss: 0.3856 - mrcnn_mask_loss: 0.33132025-05-20 02:30:10.063547: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 11s - loss: 1.2891 - rpn_class_loss: 0.0244 - rpn_bbox_loss: 0.5132 - mrcnn_class_loss: 0.0368 - mrcnn_bbox_loss: 0.3829 - mrcnn_mask_loss: 0.33182025-05-20 02:30:10.348796: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:10.350374: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3365 callback api events and 3365 activity events. 2025-05-20 02:30:10.518161: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_10 2025-05-20 02:30:10.578765: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_10/marlene.trace.json.gz 2025-05-20 02:30:10.650011: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.619 ms 2025-05-20 02:30:10.700528: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_10Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_10/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_10/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_10/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_10/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 11s - loss: 1.3243 - rpn_class_loss: 0.0235 - rpn_bbox_loss: 0.5243 - mrcnn_class_loss: 0.0508 - mrcnn_bbox_loss: 0.3979 - mrcnn_mask_loss: 0.32782025-05-20 02:30:11.008854: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 10s - loss: 1.3525 - rpn_class_loss: 0.0228 - rpn_bbox_loss: 0.5325 - mrcnn_class_loss: 0.0628 - mrcnn_bbox_loss: 0.4102 - mrcnn_mask_loss: 0.32412025-05-20 02:30:11.290731: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:11.292698: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3445 callback api events and 3445 activity events. 2025-05-20 02:30:11.455809: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_11 2025-05-20 02:30:11.532401: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_11/marlene.trace.json.gz 2025-05-20 02:30:11.593587: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.105 ms 2025-05-20 02:30:11.623577: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_11Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_11/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_11/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_11/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_11/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 10s - loss: 1.3360 - rpn_class_loss: 0.0243 - rpn_bbox_loss: 0.5154 - mrcnn_class_loss: 0.0612 - mrcnn_bbox_loss: 0.4146 - mrcnn_mask_loss: 0.32052025-05-20 02:30:11.841974: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 9s - loss: 1.3081 - rpn_class_loss: 0.0246 - rpn_bbox_loss: 0.4903 - mrcnn_class_loss: 0.0588 - mrcnn_bbox_loss: 0.4111 - mrcnn_mask_loss: 0.3233 2025-05-20 02:30:12.152995: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:12.153754: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3365 callback api events and 3365 activity events. 2025-05-20 02:30:12.294989: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_12 2025-05-20 02:30:12.371050: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_12/marlene.trace.json.gz 2025-05-20 02:30:12.495299: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.158 ms 2025-05-20 02:30:12.542392: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_12/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 9s - loss: 1.3047 - rpn_class_loss: 0.0258 - rpn_bbox_loss: 0.4933 - mrcnn_class_loss: 0.0582 - mrcnn_bbox_loss: 0.4056 - mrcnn_mask_loss: 0.32182025-05-20 02:30:12.821487: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 9s - loss: 1.2714 - rpn_class_loss: 0.0260 - rpn_bbox_loss: 0.4716 - mrcnn_class_loss: 0.0562 - mrcnn_bbox_loss: 0.3940 - mrcnn_mask_loss: 0.32362025-05-20 02:30:13.070817: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:13.071986: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3365 callback api events and 3365 activity events. 2025-05-20 02:30:13.249407: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_13 2025-05-20 02:30:13.336839: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_13/marlene.trace.json.gz 2025-05-20 02:30:13.403258: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.16 ms 2025-05-20 02:30:13.435683: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_13Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_13/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_13/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_13/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_13/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 8s - loss: 1.2700 - rpn_class_loss: 0.0255 - rpn_bbox_loss: 0.4752 - mrcnn_class_loss: 0.0564 - mrcnn_bbox_loss: 0.3955 - mrcnn_mask_loss: 0.31742025-05-20 02:30:13.725913: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 8s - loss: 1.2627 - rpn_class_loss: 0.0248 - rpn_bbox_loss: 0.4821 - mrcnn_class_loss: 0.0549 - mrcnn_bbox_loss: 0.3862 - mrcnn_mask_loss: 0.31452025-05-20 02:30:14.002484: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:14.003782: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3403 callback api events and 3403 activity events. 2025-05-20 02:30:14.167566: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_14 2025-05-20 02:30:14.259505: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_14/marlene.trace.json.gz 2025-05-20 02:30:14.344536: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.397 ms 2025-05-20 02:30:14.389293: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_14Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_14/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_14/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_14/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_14/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 7s - loss: 1.2469 - rpn_class_loss: 0.0242 - rpn_bbox_loss: 0.4727 - mrcnn_class_loss: 0.0556 - mrcnn_bbox_loss: 0.3868 - mrcnn_mask_loss: 0.30762025-05-20 02:30:14.674212: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 7s - loss: 1.2441 - rpn_class_loss: 0.0236 - rpn_bbox_loss: 0.4794 - mrcnn_class_loss: 0.0541 - mrcnn_bbox_loss: 0.3805 - mrcnn_mask_loss: 0.30652025-05-20 02:30:14.974297: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:14.979464: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3366 callback api events and 3366 activity events. 2025-05-20 02:30:15.178891: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15 2025-05-20 02:30:15.254546: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.trace.json.gz 2025-05-20 02:30:15.305586: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.23 ms 2025-05-20 02:30:15.338499: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 6s - loss: 1.2321 - rpn_class_loss: 0.0227 - rpn_bbox_loss: 0.4643 - mrcnn_class_loss: 0.0554 - mrcnn_bbox_loss: 0.3847 - mrcnn_mask_loss: 0.30502025-05-20 02:30:15.588601: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 6s - loss: 1.2179 - rpn_class_loss: 0.0218 - rpn_bbox_loss: 0.4501 - mrcnn_class_loss: 0.0550 - mrcnn_bbox_loss: 0.3874 - mrcnn_mask_loss: 0.30362025-05-20 02:30:15.880173: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:15.881401: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3404 callback api events and 3404 activity events. 2025-05-20 02:30:16.060382: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15 2025-05-20 02:30:16.170784: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.trace.json.gz 2025-05-20 02:30:16.234873: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.113 ms 2025-05-20 02:30:16.270041: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_15/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 6s - loss: 1.2020 - rpn_class_loss: 0.0219 - rpn_bbox_loss: 0.4437 - mrcnn_class_loss: 0.0533 - mrcnn_bbox_loss: 0.3823 - mrcnn_mask_loss: 0.30082025-05-20 02:30:16.500906: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 5s - loss: 1.1720 - rpn_class_loss: 0.0224 - rpn_bbox_loss: 0.4396 - mrcnn_class_loss: 0.0514 - mrcnn_bbox_loss: 0.3687 - mrcnn_mask_loss: 0.29002025-05-20 02:30:16.769748: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:16.770647: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3258 callback api events and 3258 activity events. 2025-05-20 02:30:16.962203: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_16 2025-05-20 02:30:17.053146: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_16/marlene.trace.json.gz 2025-05-20 02:30:17.122681: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.284 ms 2025-05-20 02:30:17.161116: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_16/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 5s - loss: 1.1440 - rpn_class_loss: 0.0239 - rpn_bbox_loss: 0.4346 - mrcnn_class_loss: 0.0496 - mrcnn_bbox_loss: 0.3560 - mrcnn_mask_loss: 0.28002025-05-20 02:30:17.404815: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 4s - loss: 1.1165 - rpn_class_loss: 0.0253 - rpn_bbox_loss: 0.4285 - mrcnn_class_loss: 0.0480 - mrcnn_bbox_loss: 0.3441 - mrcnn_mask_loss: 0.27072025-05-20 02:30:17.661791: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:17.662711: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3258 callback api events and 3258 activity events. 2025-05-20 02:30:17.851893: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_17 2025-05-20 02:30:17.944878: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_17/marlene.trace.json.gz 2025-05-20 02:30:17.998387: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.291 ms 2025-05-20 02:30:18.026858: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_17Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_17/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_17/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_17/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_17/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 4s - loss: 1.1359 - rpn_class_loss: 0.0247 - rpn_bbox_loss: 0.4495 - mrcnn_class_loss: 0.0476 - mrcnn_bbox_loss: 0.3460 - mrcnn_mask_loss: 0.26812025-05-20 02:30:18.255734: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 3s - loss: 1.1224 - rpn_class_loss: 0.0247 - rpn_bbox_loss: 0.4388 - mrcnn_class_loss: 0.0462 - mrcnn_bbox_loss: 0.3459 - mrcnn_mask_loss: 0.26672025-05-20 02:30:18.524085: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:18.524992: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3365 callback api events and 3365 activity events. 2025-05-20 02:30:18.678719: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_18 2025-05-20 02:30:18.744203: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_18/marlene.trace.json.gz 2025-05-20 02:30:18.796896: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.212 ms 2025-05-20 02:30:18.824345: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_18Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_18/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_18/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_18/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_18/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 3s - loss: 1.1360 - rpn_class_loss: 0.0241 - rpn_bbox_loss: 0.4496 - mrcnn_class_loss: 0.0458 - mrcnn_bbox_loss: 0.3512 - mrcnn_mask_loss: 0.26542025-05-20 02:30:19.052291: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 2s - loss: 1.1527 - rpn_class_loss: 0.0239 - rpn_bbox_loss: 0.4554 - mrcnn_class_loss: 0.0457 - mrcnn_bbox_loss: 0.3621 - mrcnn_mask_loss: 0.26562025-05-20 02:30:19.308313: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:19.309098: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3404 callback api events and 3404 activity events. 2025-05-20 02:30:19.549310: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_19 2025-05-20 02:30:19.667400: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_19/marlene.trace.json.gz 2025-05-20 02:30:19.741192: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.774 ms 2025-05-20 02:30:19.774330: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_19Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_19/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_19/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_19/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_19/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 2s - loss: 1.1353 - rpn_class_loss: 0.0237 - rpn_bbox_loss: 0.4466 - mrcnn_class_loss: 0.0447 - mrcnn_bbox_loss: 0.3561 - mrcnn_mask_loss: 0.26422025-05-20 02:30:19.987669: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 1s - loss: 1.1424 - rpn_class_loss: 0.0235 - rpn_bbox_loss: 0.4511 - mrcnn_class_loss: 0.0446 - mrcnn_bbox_loss: 0.3590 - mrcnn_mask_loss: 0.26422025-05-20 02:30:20.284788: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:20.285608: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3403 callback api events and 3403 activity events. 2025-05-20 02:30:20.499666: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_20 2025-05-20 02:30:20.567360: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_20/marlene.trace.json.gz 2025-05-20 02:30:20.628106: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.522 ms 2025-05-20 02:30:20.666830: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_20Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_20/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_20/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_20/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_20/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 1s - loss: 1.1397 - rpn_class_loss: 0.0238 - rpn_bbox_loss: 0.4466 - mrcnn_class_loss: 0.0445 - mrcnn_bbox_loss: 0.3631 - mrcnn_mask_loss: 0.26182025-05-20 02:30:20.915493: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 0s - loss: 1.1358 - rpn_class_loss: 0.0238 - rpn_bbox_loss: 0.4453 - mrcnn_class_loss: 0.0446 - mrcnn_bbox_loss: 0.3637 - mrcnn_mask_loss: 0.25852025-05-20 02:30:21.303734: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:21.304969: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3442 callback api events and 3442 activity events. 2025-05-20 02:30:21.519503: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_21 2025-05-20 02:30:21.602151: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_21/marlene.trace.json.gz 2025-05-20 02:30:21.654939: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.022 ms 2025-05-20 02:30:21.679859: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_21Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_21/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_21/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_21/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_21/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 0s - loss: 1.1294 - rpn_class_loss: 0.0236 - rpn_bbox_loss: 0.4419 - mrcnn_class_loss: 0.0443 - mrcnn_bbox_loss: 0.3633 - mrcnn_mask_loss: 0.25632025-05-20 02:30:21.892558: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:30:22.166778: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:30:22.167679: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3404 callback api events and 3405 activity events. 2025-05-20 02:30:22.409923: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_22 2025-05-20 02:30:22.481905: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_22/marlene.trace.json.gz 2025-05-20 02:30:22.529819: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.973 ms 2025-05-20 02:30:22.554496: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_22Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_22/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_22/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_22/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_30_22/marlene.kernel_stats.pb 40/40 [==============================] - 19s 486ms/step - loss: 1.1280 - rpn_class_loss: 0.0236 - rpn_bbox_loss: 0.4410 - mrcnn_class_loss: 0.0442 - mrcnn_bbox_loss: 0.3653 - mrcnn_mask_loss: 0.2539 - val_loss: 0.6210 - val_rpn_class_loss: 0.0112 - val_rpn_bbox_loss: 0.1058 - val_mrcnn_class_loss: 0.0348 - val_mrcnn_bbox_loss: 0.3120 - val_mrcnn_mask_loss: 0.1573 2025-05-20 02:30:24.828245: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:30:24.855499: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed Training all layers Starting at epoch 2. LR=0.0005 Checkpoint Path: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/mask_rcnn_learn_mask_pancarte_200525_2_{epoch:04d}.h5 Selecting layers to train conv1 (Conv2D) bn_conv1 (BatchNorm) res2a_branch2a (Conv2D) bn2a_branch2a (BatchNorm) res2a_branch2b (Conv2D) bn2a_branch2b (BatchNorm) res2a_branch2c (Conv2D) res2a_branch1 (Conv2D) bn2a_branch2c (BatchNorm) bn2a_branch1 (BatchNorm) res2b_branch2a (Conv2D) bn2b_branch2a (BatchNorm) res2b_branch2b (Conv2D) bn2b_branch2b (BatchNorm) res2b_branch2c (Conv2D) bn2b_branch2c (BatchNorm) res2c_branch2a (Conv2D) bn2c_branch2a (BatchNorm) res2c_branch2b (Conv2D) bn2c_branch2b (BatchNorm) res2c_branch2c (Conv2D) bn2c_branch2c (BatchNorm) res3a_branch2a (Conv2D) bn3a_branch2a (BatchNorm) res3a_branch2b (Conv2D) bn3a_branch2b (BatchNorm) res3a_branch2c (Conv2D) res3a_branch1 (Conv2D) bn3a_branch2c (BatchNorm) bn3a_branch1 (BatchNorm) res3b_branch2a (Conv2D) bn3b_branch2a (BatchNorm) res3b_branch2b (Conv2D) bn3b_branch2b (BatchNorm) res3b_branch2c (Conv2D) bn3b_branch2c (BatchNorm) res3c_branch2a (Conv2D) bn3c_branch2a (BatchNorm) res3c_branch2b (Conv2D) bn3c_branch2b (BatchNorm) res3c_branch2c (Conv2D) bn3c_branch2c (BatchNorm) res3d_branch2a (Conv2D) bn3d_branch2a (BatchNorm) res3d_branch2b (Conv2D) bn3d_branch2b (BatchNorm) res3d_branch2c (Conv2D) bn3d_branch2c (BatchNorm) res4a_branch2a (Conv2D) bn4a_branch2a (BatchNorm) res4a_branch2b (Conv2D) bn4a_branch2b (BatchNorm) res4a_branch2c (Conv2D) res4a_branch1 (Conv2D) bn4a_branch2c (BatchNorm) bn4a_branch1 (BatchNorm) res4b_branch2a (Conv2D) bn4b_branch2a (BatchNorm) res4b_branch2b (Conv2D) bn4b_branch2b (BatchNorm) res4b_branch2c (Conv2D) bn4b_branch2c (BatchNorm) res4c_branch2a (Conv2D) bn4c_branch2a (BatchNorm) res4c_branch2b (Conv2D) bn4c_branch2b (BatchNorm) res4c_branch2c (Conv2D) bn4c_branch2c (BatchNorm) res4d_branch2a (Conv2D) bn4d_branch2a (BatchNorm) res4d_branch2b (Conv2D) bn4d_branch2b (BatchNorm) res4d_branch2c (Conv2D) bn4d_branch2c (BatchNorm) res4e_branch2a (Conv2D) bn4e_branch2a (BatchNorm) res4e_branch2b (Conv2D) bn4e_branch2b (BatchNorm) res4e_branch2c (Conv2D) bn4e_branch2c (BatchNorm) res4f_branch2a (Conv2D) bn4f_branch2a (BatchNorm) res4f_branch2b (Conv2D) bn4f_branch2b (BatchNorm) res4f_branch2c (Conv2D) bn4f_branch2c (BatchNorm) res4g_branch2a (Conv2D) bn4g_branch2a (BatchNorm) res4g_branch2b (Conv2D) bn4g_branch2b (BatchNorm) res4g_branch2c (Conv2D) bn4g_branch2c (BatchNorm) res4h_branch2a (Conv2D) bn4h_branch2a (BatchNorm) res4h_branch2b (Conv2D) bn4h_branch2b (BatchNorm) res4h_branch2c (Conv2D) bn4h_branch2c (BatchNorm) res4i_branch2a (Conv2D) bn4i_branch2a (BatchNorm) res4i_branch2b (Conv2D) bn4i_branch2b (BatchNorm) res4i_branch2c (Conv2D) bn4i_branch2c (BatchNorm) res4j_branch2a (Conv2D) bn4j_branch2a (BatchNorm) res4j_branch2b (Conv2D) bn4j_branch2b (BatchNorm) res4j_branch2c (Conv2D) bn4j_branch2c (BatchNorm) res4k_branch2a (Conv2D) bn4k_branch2a (BatchNorm) res4k_branch2b (Conv2D) bn4k_branch2b (BatchNorm) res4k_branch2c (Conv2D) bn4k_branch2c (BatchNorm) res4l_branch2a (Conv2D) bn4l_branch2a (BatchNorm) res4l_branch2b (Conv2D) bn4l_branch2b (BatchNorm) res4l_branch2c (Conv2D) bn4l_branch2c (BatchNorm) res4m_branch2a (Conv2D) bn4m_branch2a (BatchNorm) res4m_branch2b (Conv2D) bn4m_branch2b (BatchNorm) res4m_branch2c (Conv2D) bn4m_branch2c (BatchNorm) res4n_branch2a (Conv2D) bn4n_branch2a (BatchNorm) res4n_branch2b (Conv2D) bn4n_branch2b (BatchNorm) res4n_branch2c (Conv2D) bn4n_branch2c (BatchNorm) res4o_branch2a (Conv2D) bn4o_branch2a (BatchNorm) res4o_branch2b (Conv2D) bn4o_branch2b (BatchNorm) res4o_branch2c (Conv2D) bn4o_branch2c (BatchNorm) res4p_branch2a (Conv2D) bn4p_branch2a (BatchNorm) res4p_branch2b (Conv2D) bn4p_branch2b (BatchNorm) res4p_branch2c (Conv2D) bn4p_branch2c (BatchNorm) res4q_branch2a (Conv2D) bn4q_branch2a (BatchNorm) res4q_branch2b (Conv2D) bn4q_branch2b (BatchNorm) res4q_branch2c (Conv2D) bn4q_branch2c (BatchNorm) res4r_branch2a (Conv2D) bn4r_branch2a (BatchNorm) res4r_branch2b (Conv2D) bn4r_branch2b (BatchNorm) res4r_branch2c (Conv2D) bn4r_branch2c (BatchNorm) res4s_branch2a (Conv2D) bn4s_branch2a (BatchNorm) res4s_branch2b (Conv2D) bn4s_branch2b (BatchNorm) res4s_branch2c (Conv2D) bn4s_branch2c (BatchNorm) res4t_branch2a (Conv2D) bn4t_branch2a (BatchNorm) res4t_branch2b (Conv2D) bn4t_branch2b (BatchNorm) res4t_branch2c (Conv2D) bn4t_branch2c (BatchNorm) res4u_branch2a (Conv2D) bn4u_branch2a (BatchNorm) res4u_branch2b (Conv2D) bn4u_branch2b (BatchNorm) res4u_branch2c (Conv2D) bn4u_branch2c (BatchNorm) res4v_branch2a (Conv2D) bn4v_branch2a (BatchNorm) res4v_branch2b (Conv2D) bn4v_branch2b (BatchNorm) res4v_branch2c (Conv2D) bn4v_branch2c (BatchNorm) res4w_branch2a (Conv2D) bn4w_branch2a (BatchNorm) res4w_branch2b (Conv2D) bn4w_branch2b (BatchNorm) res4w_branch2c (Conv2D) bn4w_branch2c (BatchNorm) res5a_branch2a (Conv2D) bn5a_branch2a (BatchNorm) res5a_branch2b (Conv2D) bn5a_branch2b (BatchNorm) res5a_branch2c (Conv2D) res5a_branch1 (Conv2D) bn5a_branch2c (BatchNorm) bn5a_branch1 (BatchNorm) res5b_branch2a (Conv2D) bn5b_branch2a (BatchNorm) res5b_branch2b (Conv2D) bn5b_branch2b (BatchNorm) res5b_branch2c (Conv2D) bn5b_branch2c (BatchNorm) res5c_branch2a (Conv2D) bn5c_branch2a (BatchNorm) res5c_branch2b (Conv2D) bn5c_branch2b (BatchNorm) res5c_branch2c (Conv2D) bn5c_branch2c (BatchNorm) fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.287274). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.259447). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.196029). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.212289). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.209714). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.186344). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.186489). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/engine/training_generator.py:48: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class. warnings.warn( WARNING:tensorflow:Model failed to serialize as JSON. Ignoring... cannot pickle '_thread.RLock' object mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Epoch 3/6 1/40 [..............................] - ETA: 1:20:41 - loss: 1.1350 - rpn_class_loss: 0.0213 - rpn_bbox_loss: 0.6321 - mrcnn_class_loss: 0.0012 - mrcnn_bbox_loss: 0.3575 - mrcnn_mask_loss: 0.12282025-05-20 02:34:21.624182: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 39:33 - loss: 0.9858 - rpn_class_loss: 0.0225 - rpn_bbox_loss: 0.6029 - mrcnn_class_loss: 0.0028 - mrcnn_bbox_loss: 0.2518 - mrcnn_mask_loss: 0.1058 2025-05-20 02:34:22.407862: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:22.421753: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9942 callback api events and 9943 activity events. 2025-05-20 02:34:24.288687: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_23 2025-05-20 02:34:24.793074: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_23/marlene.trace.json.gz 2025-05-20 02:34:25.307859: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.312 ms 2025-05-20 02:34:25.658197: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_23/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 26:29 - loss: 1.2801 - rpn_class_loss: 0.0254 - rpn_bbox_loss: 0.5933 - mrcnn_class_loss: 0.1012 - mrcnn_bbox_loss: 0.3471 - mrcnn_mask_loss: 0.21302025-05-20 02:34:26.368380: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 19:26 - loss: 1.2933 - rpn_class_loss: 0.0235 - rpn_bbox_loss: 0.4831 - mrcnn_class_loss: 0.1325 - mrcnn_bbox_loss: 0.3762 - mrcnn_mask_loss: 0.27812025-05-20 02:34:27.247057: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:27.252325: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10022 callback api events and 10023 activity events. 2025-05-20 02:34:28.527994: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_28 2025-05-20 02:34:28.727010: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_28/marlene.trace.json.gz 2025-05-20 02:34:29.134917: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.01 ms 2025-05-20 02:34:29.352004: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_28Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_28/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_28/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_28/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_28/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 15:27 - loss: 1.1553 - rpn_class_loss: 0.0231 - rpn_bbox_loss: 0.5028 - mrcnn_class_loss: 0.1060 - mrcnn_bbox_loss: 0.3009 - mrcnn_mask_loss: 0.22252025-05-20 02:34:29.962831: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 12:34 - loss: 1.0404 - rpn_class_loss: 0.0226 - rpn_bbox_loss: 0.4671 - mrcnn_class_loss: 0.0887 - mrcnn_bbox_loss: 0.2656 - mrcnn_mask_loss: 0.19642025-05-20 02:34:30.667353: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:30.675125: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9904 callback api events and 9905 activity events. 2025-05-20 02:34:31.834848: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_31 2025-05-20 02:34:32.356590: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_31/marlene.trace.json.gz 2025-05-20 02:34:33.057838: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 10.513 ms 2025-05-20 02:34:33.547882: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_31Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_31/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_31/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_31/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_31/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 10:46 - loss: 0.9797 - rpn_class_loss: 0.0206 - rpn_bbox_loss: 0.4077 - mrcnn_class_loss: 0.0800 - mrcnn_bbox_loss: 0.2571 - mrcnn_mask_loss: 0.21432025-05-20 02:34:34.646626: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 9:12 - loss: 0.9525 - rpn_class_loss: 0.0193 - rpn_bbox_loss: 0.3761 - mrcnn_class_loss: 0.0734 - mrcnn_bbox_loss: 0.2634 - mrcnn_mask_loss: 0.2203 2025-05-20 02:34:35.651544: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:35.657267: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9982 activity events. 2025-05-20 02:34:37.039755: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_36 2025-05-20 02:34:37.384579: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_36/marlene.trace.json.gz 2025-05-20 02:34:37.815089: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.385 ms 2025-05-20 02:34:38.021011: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_36Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_36/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_36/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_36/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_36/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 8:06 - loss: 1.0763 - rpn_class_loss: 0.0175 - rpn_bbox_loss: 0.4502 - mrcnn_class_loss: 0.0706 - mrcnn_bbox_loss: 0.3157 - mrcnn_mask_loss: 0.22232025-05-20 02:34:38.752260: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 7:05 - loss: 1.1343 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4891 - mrcnn_class_loss: 0.0656 - mrcnn_bbox_loss: 0.3415 - mrcnn_mask_loss: 0.22202025-05-20 02:34:39.372895: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:39.382414: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10020 activity events. 2025-05-20 02:34:40.571739: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_40 2025-05-20 02:34:41.223546: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_40/marlene.trace.json.gz 2025-05-20 02:34:41.874055: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.195 ms 2025-05-20 02:34:42.203208: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_40Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_40/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_40/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_40/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_40/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 6:24 - loss: 1.0776 - rpn_class_loss: 0.0158 - rpn_bbox_loss: 0.4619 - mrcnn_class_loss: 0.0620 - mrcnn_bbox_loss: 0.3186 - mrcnn_mask_loss: 0.21932025-05-20 02:34:43.176740: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 5:42 - loss: 1.0304 - rpn_class_loss: 0.0155 - rpn_bbox_loss: 0.4360 - mrcnn_class_loss: 0.0603 - mrcnn_bbox_loss: 0.3010 - mrcnn_mask_loss: 0.21772025-05-20 02:34:44.172632: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:44.184539: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9982 activity events. 2025-05-20 02:34:45.609033: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_45 2025-05-20 02:34:45.851600: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_45/marlene.trace.json.gz 2025-05-20 02:34:46.292254: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.315 ms 2025-05-20 02:34:46.425452: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_45Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_45/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_45/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_45/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_45/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 5:10 - loss: 1.0056 - rpn_class_loss: 0.0154 - rpn_bbox_loss: 0.4230 - mrcnn_class_loss: 0.0586 - mrcnn_bbox_loss: 0.2945 - mrcnn_mask_loss: 0.21412025-05-20 02:34:46.890540: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 4:38 - loss: 0.9796 - rpn_class_loss: 0.0152 - rpn_bbox_loss: 0.4093 - mrcnn_class_loss: 0.0571 - mrcnn_bbox_loss: 0.2873 - mrcnn_mask_loss: 0.21072025-05-20 02:34:47.589633: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:47.595548: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9983 activity events. 2025-05-20 02:34:48.990193: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_48 2025-05-20 02:34:49.552775: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_48/marlene.trace.json.gz 2025-05-20 02:34:49.976145: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.965 ms 2025-05-20 02:34:50.293009: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_48Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_48/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_48/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_48/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_48/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 4:15 - loss: 0.9741 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.4010 - mrcnn_class_loss: 0.0555 - mrcnn_bbox_loss: 0.2891 - mrcnn_mask_loss: 0.21372025-05-20 02:34:50.907876: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 3:51 - loss: 0.9590 - rpn_class_loss: 0.0144 - rpn_bbox_loss: 0.3935 - mrcnn_class_loss: 0.0542 - mrcnn_bbox_loss: 0.2818 - mrcnn_mask_loss: 0.21522025-05-20 02:34:51.649754: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:51.660719: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9980 activity events. 2025-05-20 02:34:53.040236: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_52 2025-05-20 02:34:53.365698: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_52/marlene.trace.json.gz 2025-05-20 02:34:53.973991: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.395 ms 2025-05-20 02:34:54.252556: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_52Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_52/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_52/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_52/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_52/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 3:33 - loss: 0.9372 - rpn_class_loss: 0.0143 - rpn_bbox_loss: 0.3811 - mrcnn_class_loss: 0.0576 - mrcnn_bbox_loss: 0.2739 - mrcnn_mask_loss: 0.21032025-05-20 02:34:55.151601: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 3:13 - loss: 0.9109 - rpn_class_loss: 0.0140 - rpn_bbox_loss: 0.3646 - mrcnn_class_loss: 0.0567 - mrcnn_bbox_loss: 0.2695 - mrcnn_mask_loss: 0.2061/usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 2025-05-20 02:34:55.981865: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:55.988450: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9980 activity events. 2025-05-20 02:34:56.706021: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_56 2025-05-20 02:34:57.110548: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_56/marlene.trace.json.gz 2025-05-20 02:34:57.646918: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.683 ms 2025-05-20 02:34:58.032409: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_56Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_56/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_56/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_56/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_34_56/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 2:58 - loss: 0.8873 - rpn_class_loss: 0.0142 - rpn_bbox_loss: 0.3473 - mrcnn_class_loss: 0.0549 - mrcnn_bbox_loss: 0.2674 - mrcnn_mask_loss: 0.20342025-05-20 02:34:58.719269: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 2:42 - loss: 0.8656 - rpn_class_loss: 0.0142 - rpn_bbox_loss: 0.3328 - mrcnn_class_loss: 0.0539 - mrcnn_bbox_loss: 0.2648 - mrcnn_mask_loss: 0.20002025-05-20 02:34:59.599744: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:34:59.609036: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:35:00.566681: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_00 2025-05-20 02:35:00.901761: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_00/marlene.trace.json.gz 2025-05-20 02:35:01.248206: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.402 ms 2025-05-20 02:35:01.518877: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_00Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_00/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_00/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_00/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_00/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 2:29 - loss: 0.8744 - rpn_class_loss: 0.0141 - rpn_bbox_loss: 0.3412 - mrcnn_class_loss: 0.0527 - mrcnn_bbox_loss: 0.2655 - mrcnn_mask_loss: 0.20092025-05-20 02:35:02.154938: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 22/40 [===============>..............] - ETA: 2:15 - loss: 0.8689 - rpn_class_loss: 0.0138 - rpn_bbox_loss: 0.3422 - mrcnn_class_loss: 0.0514 - mrcnn_bbox_loss: 0.2606 - mrcnn_mask_loss: 0.20082025-05-20 02:35:02.839576: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:02.843061: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9943 callback api events and 9943 activity events. 2025-05-20 02:35:03.985351: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_03 2025-05-20 02:35:04.582592: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_03/marlene.trace.json.gz 2025-05-20 02:35:05.240239: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 9.093 ms 2025-05-20 02:35:05.607411: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_03Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_03/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_03/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_03/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_03/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 2:04 - loss: 0.8878 - rpn_class_loss: 0.0134 - rpn_bbox_loss: 0.3670 - mrcnn_class_loss: 0.0500 - mrcnn_bbox_loss: 0.2557 - mrcnn_mask_loss: 0.20172025-05-20 02:35:06.553561: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 1:53 - loss: 0.8985 - rpn_class_loss: 0.0130 - rpn_bbox_loss: 0.3853 - mrcnn_class_loss: 0.0487 - mrcnn_bbox_loss: 0.2500 - mrcnn_mask_loss: 0.20152025-05-20 02:35:07.598056: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:07.625182: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9943 callback api events and 9943 activity events. 2025-05-20 02:35:09.429450: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_08 2025-05-20 02:35:09.904181: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_08/marlene.trace.json.gz 2025-05-20 02:35:10.534633: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 13.605 ms 2025-05-20 02:35:10.916898: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_08Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_08/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_08/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_08/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_08/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 1:44 - loss: 0.8954 - rpn_class_loss: 0.0128 - rpn_bbox_loss: 0.3828 - mrcnn_class_loss: 0.0487 - mrcnn_bbox_loss: 0.2520 - mrcnn_mask_loss: 0.19902025-05-20 02:35:12.024974: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 1:34 - loss: 0.8952 - rpn_class_loss: 0.0125 - rpn_bbox_loss: 0.3862 - mrcnn_class_loss: 0.0479 - mrcnn_bbox_loss: 0.2513 - mrcnn_mask_loss: 0.19742025-05-20 02:35:13.161674: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:13.172636: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9980 activity events. 2025-05-20 02:35:14.604561: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_14 2025-05-20 02:35:14.835909: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_14/marlene.trace.json.gz 2025-05-20 02:35:15.274255: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.484 ms 2025-05-20 02:35:15.607799: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_14Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_14/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_14/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_14/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_14/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 1:26 - loss: 0.9776 - rpn_class_loss: 0.0120 - rpn_bbox_loss: 0.4304 - mrcnn_class_loss: 0.0521 - mrcnn_bbox_loss: 0.2884 - mrcnn_mask_loss: 0.19462025-05-20 02:35:16.206747: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 1:17 - loss: 1.0483 - rpn_class_loss: 0.0116 - rpn_bbox_loss: 0.4676 - mrcnn_class_loss: 0.0563 - mrcnn_bbox_loss: 0.3208 - mrcnn_mask_loss: 0.19192025-05-20 02:35:17.416509: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:17.422507: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9980 activity events. 2025-05-20 02:35:18.422340: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_17 2025-05-20 02:35:18.825207: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_17/marlene.trace.json.gz 2025-05-20 02:35:19.442222: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 11.455 ms 2025-05-20 02:35:19.701607: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_17Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_17/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_17/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_17/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_17/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 1:09 - loss: 1.0432 - rpn_class_loss: 0.0113 - rpn_bbox_loss: 0.4642 - mrcnn_class_loss: 0.0572 - mrcnn_bbox_loss: 0.3225 - mrcnn_mask_loss: 0.18802025-05-20 02:35:20.511326: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 1:01 - loss: 1.0357 - rpn_class_loss: 0.0111 - rpn_bbox_loss: 0.4606 - mrcnn_class_loss: 0.0566 - mrcnn_bbox_loss: 0.3229 - mrcnn_mask_loss: 0.18462025-05-20 02:35:21.476173: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:21.488288: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10022 activity events. 2025-05-20 02:35:22.795889: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_22 2025-05-20 02:35:23.062802: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_22/marlene.trace.json.gz 2025-05-20 02:35:23.555910: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 12.224 ms 2025-05-20 02:35:23.964356: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_22Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_22/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_22/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_22/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_22/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 54s - loss: 1.0196 - rpn_class_loss: 0.0111 - rpn_bbox_loss: 0.4507 - mrcnn_class_loss: 0.0566 - mrcnn_bbox_loss: 0.3188 - mrcnn_mask_loss: 0.1824 2025-05-20 02:35:24.870208: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 47s - loss: 0.9995 - rpn_class_loss: 0.0110 - rpn_bbox_loss: 0.4392 - mrcnn_class_loss: 0.0558 - mrcnn_bbox_loss: 0.3134 - mrcnn_mask_loss: 0.18012025-05-20 02:35:25.932844: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:25.944772: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:35:27.106175: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_26 2025-05-20 02:35:27.393870: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_26/marlene.trace.json.gz 2025-05-20 02:35:27.699725: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.411 ms 2025-05-20 02:35:27.823606: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_26/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 40s - loss: 0.9983 - rpn_class_loss: 0.0114 - rpn_bbox_loss: 0.4408 - mrcnn_class_loss: 0.0550 - mrcnn_bbox_loss: 0.3115 - mrcnn_mask_loss: 0.17962025-05-20 02:35:28.257683: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 33s - loss: 0.9909 - rpn_class_loss: 0.0116 - rpn_bbox_loss: 0.4386 - mrcnn_class_loss: 0.0543 - mrcnn_bbox_loss: 0.3078 - mrcnn_mask_loss: 0.17862025-05-20 02:35:29.100053: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:29.108135: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9984 activity events. 2025-05-20 02:35:30.376151: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_29 2025-05-20 02:35:30.838383: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_29/marlene.trace.json.gz 2025-05-20 02:35:31.668631: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 25.563 ms 2025-05-20 02:35:31.992819: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_29Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_29/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_29/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_29/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_29/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 27s - loss: 0.9864 - rpn_class_loss: 0.0114 - rpn_bbox_loss: 0.4404 - mrcnn_class_loss: 0.0531 - mrcnn_bbox_loss: 0.3047 - mrcnn_mask_loss: 0.17692025-05-20 02:35:32.715612: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 21s - loss: 0.9871 - rpn_class_loss: 0.0112 - rpn_bbox_loss: 0.4441 - mrcnn_class_loss: 0.0520 - mrcnn_bbox_loss: 0.3046 - mrcnn_mask_loss: 0.17522025-05-20 02:35:33.585839: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:33.598396: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:35:34.466859: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_34 2025-05-20 02:35:34.997745: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_34/marlene.trace.json.gz 2025-05-20 02:35:35.443058: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.455 ms 2025-05-20 02:35:35.598570: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_34/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 16s - loss: 1.0150 - rpn_class_loss: 0.0111 - rpn_bbox_loss: 0.4735 - mrcnn_class_loss: 0.0527 - mrcnn_bbox_loss: 0.3049 - mrcnn_mask_loss: 0.17282025-05-20 02:35:36.299154: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 10s - loss: 1.0453 - rpn_class_loss: 0.0111 - rpn_bbox_loss: 0.5018 - mrcnn_class_loss: 0.0536 - mrcnn_bbox_loss: 0.3078 - mrcnn_mask_loss: 0.17112025-05-20 02:35:37.103272: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:37.112051: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10019 activity events. 2025-05-20 02:35:38.499487: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_37 2025-05-20 02:35:39.012600: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_37/marlene.trace.json.gz 2025-05-20 02:35:39.601126: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.266 ms 2025-05-20 02:35:39.855268: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_37Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_37/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_37/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_37/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_37/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 5s - loss: 1.0307 - rpn_class_loss: 0.0109 - rpn_bbox_loss: 0.4936 - mrcnn_class_loss: 0.0527 - mrcnn_bbox_loss: 0.3040 - mrcnn_mask_loss: 0.1695 2025-05-20 02:35:40.809916: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:35:41.807822: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:35:41.815915: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9984 activity events. 2025-05-20 02:35:43.257352: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_42 2025-05-20 02:35:43.958142: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_42/marlene.trace.json.gz 2025-05-20 02:35:44.541304: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 12.008 ms 2025-05-20 02:35:44.883548: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_42Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_42/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_42/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_42/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_35_42/marlene.kernel_stats.pb 40/40 [==============================] - 231s 6s/step - loss: 1.0208 - rpn_class_loss: 0.0107 - rpn_bbox_loss: 0.4896 - mrcnn_class_loss: 0.0518 - mrcnn_bbox_loss: 0.3005 - mrcnn_mask_loss: 0.1681 - val_loss: 0.4014 - val_rpn_class_loss: 0.0041 - val_rpn_bbox_loss: 0.1273 - val_mrcnn_class_loss: 0.0379 - val_mrcnn_bbox_loss: 0.1129 - val_mrcnn_mask_loss: 0.1191 Epoch 4/6 1/40 [..............................] - ETA: 19s - loss: 0.9617 - rpn_class_loss: 0.0087 - rpn_bbox_loss: 0.2111 - mrcnn_class_loss: 0.0156 - mrcnn_bbox_loss: 0.4613 - mrcnn_mask_loss: 0.26502025-05-20 02:36:11.865838: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 21s - loss: 0.8824 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.2021 - mrcnn_class_loss: 0.0204 - mrcnn_bbox_loss: 0.3972 - mrcnn_mask_loss: 0.25422025-05-20 02:36:12.596062: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:12.603370: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:36:13.277783: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_13 2025-05-20 02:36:13.552889: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_13/marlene.trace.json.gz 2025-05-20 02:36:13.873616: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.981 ms 2025-05-20 02:36:14.067726: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_13Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_13/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_13/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_13/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_13/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 40s - loss: 0.7762 - rpn_class_loss: 0.0093 - rpn_bbox_loss: 0.2145 - mrcnn_class_loss: 0.0231 - mrcnn_bbox_loss: 0.3106 - mrcnn_mask_loss: 0.21872025-05-20 02:36:14.610029: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 36s - loss: 0.7109 - rpn_class_loss: 0.0097 - rpn_bbox_loss: 0.2147 - mrcnn_class_loss: 0.0269 - mrcnn_bbox_loss: 0.2589 - mrcnn_mask_loss: 0.20082025-05-20 02:36:15.698663: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:15.727132: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9984 activity events. 2025-05-20 02:36:17.132472: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_16 2025-05-20 02:36:17.771459: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_16/marlene.trace.json.gz 2025-05-20 02:36:18.550236: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 11.274 ms 2025-05-20 02:36:18.982381: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_16/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 1:00 - loss: 0.6871 - rpn_class_loss: 0.0088 - rpn_bbox_loss: 0.2069 - mrcnn_class_loss: 0.0258 - mrcnn_bbox_loss: 0.2544 - mrcnn_mask_loss: 0.19132025-05-20 02:36:19.954875: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 52s - loss: 0.6620 - rpn_class_loss: 0.0080 - rpn_bbox_loss: 0.2003 - mrcnn_class_loss: 0.0252 - mrcnn_bbox_loss: 0.2426 - mrcnn_mask_loss: 0.1859 2025-05-20 02:36:20.872508: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:20.894055: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9984 activity events. 2025-05-20 02:36:22.233226: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_21 2025-05-20 02:36:22.638693: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_21/marlene.trace.json.gz 2025-05-20 02:36:23.170225: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.051 ms 2025-05-20 02:36:23.439133: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_21Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_21/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_21/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_21/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_21/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 59s - loss: 0.6485 - rpn_class_loss: 0.0079 - rpn_bbox_loss: 0.2030 - mrcnn_class_loss: 0.0261 - mrcnn_bbox_loss: 0.2304 - mrcnn_mask_loss: 0.18102025-05-20 02:36:23.975381: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 53s - loss: 0.6424 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 0.2102 - mrcnn_class_loss: 0.0249 - mrcnn_bbox_loss: 0.2233 - mrcnn_mask_loss: 0.17622025-05-20 02:36:24.789471: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:24.800896: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9985 activity events. 2025-05-20 02:36:25.909615: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_25 2025-05-20 02:36:26.264978: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_25/marlene.trace.json.gz 2025-05-20 02:36:26.680504: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.821 ms 2025-05-20 02:36:26.944435: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_25Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_25/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_25/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_25/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_25/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 56s - loss: 0.6518 - rpn_class_loss: 0.0072 - rpn_bbox_loss: 0.2249 - mrcnn_class_loss: 0.0241 - mrcnn_bbox_loss: 0.2250 - mrcnn_mask_loss: 0.17062025-05-20 02:36:27.746824: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 51s - loss: 0.6458 - rpn_class_loss: 0.0066 - rpn_bbox_loss: 0.2328 - mrcnn_class_loss: 0.0244 - mrcnn_bbox_loss: 0.2180 - mrcnn_mask_loss: 0.16402025-05-20 02:36:28.616539: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:28.628717: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9984 activity events. 2025-05-20 02:36:29.445615: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_29 2025-05-20 02:36:29.685016: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_29/marlene.trace.json.gz 2025-05-20 02:36:30.072379: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.072 ms 2025-05-20 02:36:30.202461: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_29Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_29/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_29/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_29/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_29/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 50s - loss: 0.7327 - rpn_class_loss: 0.0061 - rpn_bbox_loss: 0.2431 - mrcnn_class_loss: 0.0317 - mrcnn_bbox_loss: 0.2912 - mrcnn_mask_loss: 0.16052025-05-20 02:36:30.652764: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 46s - loss: 0.7795 - rpn_class_loss: 0.0056 - rpn_bbox_loss: 0.2553 - mrcnn_class_loss: 0.0357 - mrcnn_bbox_loss: 0.3264 - mrcnn_mask_loss: 0.15652025-05-20 02:36:31.682138: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:31.694226: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10019 activity events. 2025-05-20 02:36:33.276222: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_32 2025-05-20 02:36:33.792127: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_32/marlene.trace.json.gz 2025-05-20 02:36:34.273944: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 9.024 ms 2025-05-20 02:36:34.567901: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_32Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_32/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_32/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_32/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_32/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 49s - loss: 0.8174 - rpn_class_loss: 0.0053 - rpn_bbox_loss: 0.2704 - mrcnn_class_loss: 0.0354 - mrcnn_bbox_loss: 0.3515 - mrcnn_mask_loss: 0.15472025-05-20 02:36:35.231013: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 45s - loss: 0.8281 - rpn_class_loss: 0.0050 - rpn_bbox_loss: 0.2778 - mrcnn_class_loss: 0.0373 - mrcnn_bbox_loss: 0.3557 - mrcnn_mask_loss: 0.15232025-05-20 02:36:36.148973: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:36.166759: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10024 activity events. 2025-05-20 02:36:37.142245: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_36 2025-05-20 02:36:37.435862: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_36/marlene.trace.json.gz 2025-05-20 02:36:37.949280: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.086 ms 2025-05-20 02:36:38.269247: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_36Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_36/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_36/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_36/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_36/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 45s - loss: 0.8134 - rpn_class_loss: 0.0052 - rpn_bbox_loss: 0.2811 - mrcnn_class_loss: 0.0359 - mrcnn_bbox_loss: 0.3393 - mrcnn_mask_loss: 0.15192025-05-20 02:36:38.917504: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 42s - loss: 0.8027 - rpn_class_loss: 0.0054 - rpn_bbox_loss: 0.2827 - mrcnn_class_loss: 0.0356 - mrcnn_bbox_loss: 0.3279 - mrcnn_mask_loss: 0.15112025-05-20 02:36:40.359918: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:40.377956: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9943 callback api events and 9948 activity events. 2025-05-20 02:36:41.471457: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_41 2025-05-20 02:36:41.845852: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_41/marlene.trace.json.gz 2025-05-20 02:36:42.351263: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.735 ms 2025-05-20 02:36:42.690353: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_41Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_41/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_41/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_41/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_41/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 47s - loss: 0.7765 - rpn_class_loss: 0.0058 - rpn_bbox_loss: 0.2693 - mrcnn_class_loss: 0.0342 - mrcnn_bbox_loss: 0.3167 - mrcnn_mask_loss: 0.15052025-05-20 02:36:46.666191: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 48s - loss: 0.7505 - rpn_class_loss: 0.0062 - rpn_bbox_loss: 0.2570 - mrcnn_class_loss: 0.0330 - mrcnn_bbox_loss: 0.3044 - mrcnn_mask_loss: 0.14982025-05-20 02:36:51.333475: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:36:51.339054: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9943 callback api events and 9953 activity events. 2025-05-20 02:36:52.400708: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_51 2025-05-20 02:36:52.789769: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_51/marlene.trace.json.gz 2025-05-20 02:36:53.137671: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.515 ms 2025-05-20 02:36:53.383508: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_51Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_51/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_51/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_51/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_36_51/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 50s - loss: 0.7421 - rpn_class_loss: 0.0064 - rpn_bbox_loss: 0.2552 - mrcnn_class_loss: 0.0326 - mrcnn_bbox_loss: 0.3014 - mrcnn_mask_loss: 0.14652025-05-20 02:36:57.134847: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 49s - loss: 0.7334 - rpn_class_loss: 0.0062 - rpn_bbox_loss: 0.2572 - mrcnn_class_loss: 0.0319 - mrcnn_bbox_loss: 0.2947 - mrcnn_mask_loss: 0.14342025-05-20 02:37:00.961750: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:37:00.965666: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9990 activity events. 2025-05-20 02:37:01.484111: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_01 2025-05-20 02:37:01.746995: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_01/marlene.trace.json.gz 2025-05-20 02:37:02.015575: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.942 ms 2025-05-20 02:37:02.185459: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_01Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_01/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_01/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_01/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_01/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 49s - loss: 0.7931 - rpn_class_loss: 0.0069 - rpn_bbox_loss: 0.3187 - mrcnn_class_loss: 0.0310 - mrcnn_bbox_loss: 0.2975 - mrcnn_mask_loss: 0.13902025-05-20 02:37:05.518169: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 47s - loss: 0.8240 - rpn_class_loss: 0.0076 - rpn_bbox_loss: 0.3534 - mrcnn_class_loss: 0.0302 - mrcnn_bbox_loss: 0.2974 - mrcnn_mask_loss: 0.13542025-05-20 02:37:10.099777: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:37:10.107300: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9990 activity events. 2025-05-20 02:37:10.774324: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_10 2025-05-20 02:37:11.115300: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_10/marlene.trace.json.gz 2025-05-20 02:37:11.379057: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.63 ms 2025-05-20 02:37:11.526565: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_10Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_10/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_10/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_10/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_10/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 47s - loss: 0.8090 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 0.3434 - mrcnn_class_loss: 0.0297 - mrcnn_bbox_loss: 0.2939 - mrcnn_mask_loss: 0.13422025-05-20 02:37:16.185893: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 45s - loss: 0.7928 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.3346 - mrcnn_class_loss: 0.0299 - mrcnn_bbox_loss: 0.2875 - mrcnn_mask_loss: 0.13312025-05-20 02:37:19.451460: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:37:19.466721: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10024 activity events. 2025-05-20 02:37:20.357679: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_20 2025-05-20 02:37:20.833378: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_20/marlene.trace.json.gz 2025-05-20 02:37:21.235095: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.432 ms 2025-05-20 02:37:21.459828: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_20Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_20/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_20/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_20/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_20/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 46s - loss: 0.7938 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.3407 - mrcnn_class_loss: 0.0298 - mrcnn_bbox_loss: 0.2823 - mrcnn_mask_loss: 0.13332025-05-20 02:37:28.808078: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 42s - loss: 0.7867 - rpn_class_loss: 0.0075 - rpn_bbox_loss: 0.3396 - mrcnn_class_loss: 0.0298 - mrcnn_bbox_loss: 0.2765 - mrcnn_mask_loss: 0.13312025-05-20 02:37:29.780939: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:37:29.785427: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9987 activity events. 2025-05-20 02:37:30.812359: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_30 2025-05-20 02:37:31.157046: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_30/marlene.trace.json.gz 2025-05-20 02:37:31.434999: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.692 ms 2025-05-20 02:37:31.660538: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_30Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_30/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_30/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_30/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_30/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 42s - loss: 0.7819 - rpn_class_loss: 0.0073 - rpn_bbox_loss: 0.3403 - mrcnn_class_loss: 0.0306 - mrcnn_bbox_loss: 0.2715 - mrcnn_mask_loss: 0.13222025-05-20 02:37:38.672325: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 37s - loss: 0.7679 - rpn_class_loss: 0.0071 - rpn_bbox_loss: 0.3326 - mrcnn_class_loss: 0.0306 - mrcnn_bbox_loss: 0.2664 - mrcnn_mask_loss: 0.13122025-05-20 02:37:39.590745: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:37:39.600337: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:37:40.814092: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_40 2025-05-20 02:37:41.057713: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_40/marlene.trace.json.gz 2025-05-20 02:37:41.501909: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.952 ms 2025-05-20 02:37:41.765088: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_40Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_40/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_40/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_40/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_40/marlene.kernel_stats.pb /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 29/40 [====================>.........] - ETA: 37s - loss: 0.7594 - rpn_class_loss: 0.0072 - rpn_bbox_loss: 0.3279 - mrcnn_class_loss: 0.0306 - mrcnn_bbox_loss: 0.2639 - mrcnn_mask_loss: 0.12982025-05-20 02:37:48.972632: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 32s - loss: 0.7479 - rpn_class_loss: 0.0072 - rpn_bbox_loss: 0.3210 - mrcnn_class_loss: 0.0301 - mrcnn_bbox_loss: 0.2610 - mrcnn_mask_loss: 0.12852025-05-20 02:37:50.394650: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:37:50.410267: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9942 callback api events and 9948 activity events. 2025-05-20 02:37:51.435215: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_50 2025-05-20 02:37:51.784679: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_50/marlene.trace.json.gz 2025-05-20 02:37:52.025909: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.665 ms 2025-05-20 02:37:52.222602: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_50Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_50/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_50/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_50/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_37_50/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 31s - loss: 0.7475 - rpn_class_loss: 0.0072 - rpn_bbox_loss: 0.3239 - mrcnn_class_loss: 0.0295 - mrcnn_bbox_loss: 0.2595 - mrcnn_mask_loss: 0.12742025-05-20 02:37:58.614103: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 27s - loss: 0.7456 - rpn_class_loss: 0.0072 - rpn_bbox_loss: 0.3259 - mrcnn_class_loss: 0.0291 - mrcnn_bbox_loss: 0.2570 - mrcnn_mask_loss: 0.12642025-05-20 02:38:00.244750: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:38:00.250029: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10025 activity events. 2025-05-20 02:38:01.171016: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_00 2025-05-20 02:38:01.621285: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_00/marlene.trace.json.gz 2025-05-20 02:38:02.093013: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.24 ms 2025-05-20 02:38:02.318641: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_00Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_00/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_00/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_00/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_00/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 25s - loss: 0.7622 - rpn_class_loss: 0.0074 - rpn_bbox_loss: 0.3346 - mrcnn_class_loss: 0.0309 - mrcnn_bbox_loss: 0.2637 - mrcnn_mask_loss: 0.12562025-05-20 02:38:10.856812: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 21s - loss: 0.7743 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 0.3357 - mrcnn_class_loss: 0.0323 - mrcnn_bbox_loss: 0.2735 - mrcnn_mask_loss: 0.12492025-05-20 02:38:11.934425: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:38:11.939095: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10022 callback api events and 10028 activity events. 2025-05-20 02:38:12.493801: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_12 2025-05-20 02:38:12.682589: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_12/marlene.trace.json.gz 2025-05-20 02:38:12.942597: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.979 ms 2025-05-20 02:38:13.100442: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_12/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 18s - loss: 0.7665 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.3280 - mrcnn_class_loss: 0.0321 - mrcnn_bbox_loss: 0.2720 - mrcnn_mask_loss: 0.12662025-05-20 02:38:21.905119: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 14s - loss: 0.7565 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.3209 - mrcnn_class_loss: 0.0314 - mrcnn_bbox_loss: 0.2688 - mrcnn_mask_loss: 0.12782025-05-20 02:38:23.016820: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:38:23.021618: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:38:23.778450: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_23 2025-05-20 02:38:24.073121: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_23/marlene.trace.json.gz 2025-05-20 02:38:24.347686: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.698 ms 2025-05-20 02:38:24.549513: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_23/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 11s - loss: 0.7626 - rpn_class_loss: 0.0075 - rpn_bbox_loss: 0.3325 - mrcnn_class_loss: 0.0313 - mrcnn_bbox_loss: 0.2641 - mrcnn_mask_loss: 0.12732025-05-20 02:38:29.535528: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 7s - loss: 0.7670 - rpn_class_loss: 0.0073 - rpn_bbox_loss: 0.3419 - mrcnn_class_loss: 0.0312 - mrcnn_bbox_loss: 0.2594 - mrcnn_mask_loss: 0.1272 2025-05-20 02:38:32.803288: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:38:32.809999: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9993 activity events. 2025-05-20 02:38:33.866687: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_33 2025-05-20 02:38:34.071562: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_33/marlene.trace.json.gz 2025-05-20 02:38:34.361445: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.307 ms 2025-05-20 02:38:34.546599: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_33Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_33/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_33/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_33/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_33/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 3s - loss: 0.7647 - rpn_class_loss: 0.0074 - rpn_bbox_loss: 0.3451 - mrcnn_class_loss: 0.0308 - mrcnn_bbox_loss: 0.2556 - mrcnn_mask_loss: 0.12582025-05-20 02:38:39.383385: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:38:41.078941: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:38:41.083566: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9942 callback api events and 9948 activity events. 2025-05-20 02:38:42.239485: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_41 2025-05-20 02:38:42.763202: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_41/marlene.trace.json.gz 2025-05-20 02:38:43.133003: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.706 ms 2025-05-20 02:38:43.311695: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_41Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_41/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_41/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_41/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_41/marlene.kernel_stats.pb 40/40 [==============================] - 156s 4s/step - loss: 0.7611 - rpn_class_loss: 0.0076 - rpn_bbox_loss: 0.3467 - mrcnn_class_loss: 0.0305 - mrcnn_bbox_loss: 0.2518 - mrcnn_mask_loss: 0.1246 - val_loss: 0.3408 - val_rpn_class_loss: 0.0056 - val_rpn_bbox_loss: 0.0800 - val_mrcnn_class_loss: 0.0119 - val_mrcnn_bbox_loss: 0.1574 - val_mrcnn_mask_loss: 0.0858 Epoch 5/6 1/40 [..............................] - ETA: 1:02 - loss: 0.3567 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.0852 - mrcnn_class_loss: 0.0253 - mrcnn_bbox_loss: 0.1550 - mrcnn_mask_loss: 0.08352025-05-20 02:38:50.531594: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 37s - loss: 0.3005 - rpn_class_loss: 0.0062 - rpn_bbox_loss: 0.0621 - mrcnn_class_loss: 0.0183 - mrcnn_bbox_loss: 0.1294 - mrcnn_mask_loss: 0.0846 2025-05-20 02:38:51.044827: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:38:51.048263: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9943 callback api events and 9950 activity events. 2025-05-20 02:38:51.782000: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_51 2025-05-20 02:38:52.019807: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_51/marlene.trace.json.gz 2025-05-20 02:38:52.295890: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.119 ms 2025-05-20 02:38:52.506998: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_51Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_51/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_51/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_51/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_51/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 1:56 - loss: 0.3398 - rpn_class_loss: 0.0057 - rpn_bbox_loss: 0.1216 - mrcnn_class_loss: 0.0162 - mrcnn_bbox_loss: 0.1071 - mrcnn_mask_loss: 0.08902025-05-20 02:38:58.352450: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 1:29 - loss: 0.3371 - rpn_class_loss: 0.0056 - rpn_bbox_loss: 0.1022 - mrcnn_class_loss: 0.0178 - mrcnn_bbox_loss: 0.1190 - mrcnn_mask_loss: 0.09252025-05-20 02:38:59.074266: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:38:59.078696: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9943 callback api events and 9943 activity events. 2025-05-20 02:38:59.784568: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_59 2025-05-20 02:39:00.026567: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_59/marlene.trace.json.gz 2025-05-20 02:39:00.374544: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.068 ms 2025-05-20 02:39:00.564204: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_59Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_59/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_59/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_59/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_38_59/marlene.kernel_stats.pb /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 5/40 [==>...........................] - ETA: 2:04 - loss: 0.3856 - rpn_class_loss: 0.0061 - rpn_bbox_loss: 0.1129 - mrcnn_class_loss: 0.0199 - mrcnn_bbox_loss: 0.1510 - mrcnn_mask_loss: 0.09582025-05-20 02:39:06.758366: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 1:43 - loss: 0.3894 - rpn_class_loss: 0.0065 - rpn_bbox_loss: 0.1184 - mrcnn_class_loss: 0.0195 - mrcnn_bbox_loss: 0.1470 - mrcnn_mask_loss: 0.09802025-05-20 02:39:07.292178: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:39:07.296868: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9988 activity events. 2025-05-20 02:39:08.100688: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_07 2025-05-20 02:39:08.374120: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_07/marlene.trace.json.gz 2025-05-20 02:39:08.724006: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.548 ms 2025-05-20 02:39:08.915692: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_07Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_07/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_07/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_07/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_07/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 1:54 - loss: 0.4547 - rpn_class_loss: 0.0061 - rpn_bbox_loss: 0.1861 - mrcnn_class_loss: 0.0212 - mrcnn_bbox_loss: 0.1344 - mrcnn_mask_loss: 0.10692025-05-20 02:39:13.155121: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 1:41 - loss: 0.5004 - rpn_class_loss: 0.0059 - rpn_bbox_loss: 0.2341 - mrcnn_class_loss: 0.0202 - mrcnn_bbox_loss: 0.1312 - mrcnn_mask_loss: 0.10892025-05-20 02:39:14.450641: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:39:14.456675: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9988 activity events. 2025-05-20 02:39:15.166536: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_14 2025-05-20 02:39:15.379853: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_14/marlene.trace.json.gz 2025-05-20 02:39:15.717054: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.236 ms 2025-05-20 02:39:15.921419: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_14Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_14/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_14/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_14/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_14/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 1:49 - loss: 0.5617 - rpn_class_loss: 0.0057 - rpn_bbox_loss: 0.2838 - mrcnn_class_loss: 0.0190 - mrcnn_bbox_loss: 0.1423 - mrcnn_mask_loss: 0.11092025-05-20 02:39:20.631580: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 1:36 - loss: 0.6036 - rpn_class_loss: 0.0055 - rpn_bbox_loss: 0.3213 - mrcnn_class_loss: 0.0191 - mrcnn_bbox_loss: 0.1459 - mrcnn_mask_loss: 0.11172025-05-20 02:39:21.258257: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:39:21.264139: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:39:22.011330: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_21 2025-05-20 02:39:22.283538: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_21/marlene.trace.json.gz 2025-05-20 02:39:22.623854: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.168 ms 2025-05-20 02:39:22.817236: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_21Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_21/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_21/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_21/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_21/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 1:44 - loss: 0.6249 - rpn_class_loss: 0.0061 - rpn_bbox_loss: 0.3017 - mrcnn_class_loss: 0.0217 - mrcnn_bbox_loss: 0.1743 - mrcnn_mask_loss: 0.12102025-05-20 02:39:28.564921: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 1:34 - loss: 0.6118 - rpn_class_loss: 0.0071 - rpn_bbox_loss: 0.2831 - mrcnn_class_loss: 0.0214 - mrcnn_bbox_loss: 0.1746 - mrcnn_mask_loss: 0.12542025-05-20 02:39:29.513798: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:39:29.518082: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9941 callback api events and 9941 activity events. 2025-05-20 02:39:30.158467: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_29 2025-05-20 02:39:30.498012: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_29/marlene.trace.json.gz 2025-05-20 02:39:30.947207: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.465 ms 2025-05-20 02:39:31.128349: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_29Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_29/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_29/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_29/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_29/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 1:35 - loss: 0.6562 - rpn_class_loss: 0.0066 - rpn_bbox_loss: 0.2856 - mrcnn_class_loss: 0.0371 - mrcnn_bbox_loss: 0.2037 - mrcnn_mask_loss: 0.12322025-05-20 02:39:35.131074: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 1:26 - loss: 0.6871 - rpn_class_loss: 0.0061 - rpn_bbox_loss: 0.2851 - mrcnn_class_loss: 0.0495 - mrcnn_bbox_loss: 0.2249 - mrcnn_mask_loss: 0.12142025-05-20 02:39:35.811942: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:39:35.819668: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:39:36.507292: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_36 2025-05-20 02:39:36.756103: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_36/marlene.trace.json.gz 2025-05-20 02:39:37.028071: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.62 ms 2025-05-20 02:39:37.222984: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_36Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_36/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_36/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_36/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_36/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 1:27 - loss: 0.6825 - rpn_class_loss: 0.0058 - rpn_bbox_loss: 0.2700 - mrcnn_class_loss: 0.0491 - mrcnn_bbox_loss: 0.2386 - mrcnn_mask_loss: 0.11902025-05-20 02:39:41.458086: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 1:22 - loss: 0.6759 - rpn_class_loss: 0.0054 - rpn_bbox_loss: 0.2569 - mrcnn_class_loss: 0.0475 - mrcnn_bbox_loss: 0.2496 - mrcnn_mask_loss: 0.11652025-05-20 02:39:43.952187: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:39:43.956999: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9997 activity events. 2025-05-20 02:39:44.691436: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_44 2025-05-20 02:39:44.865801: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_44/marlene.trace.json.gz 2025-05-20 02:39:45.112480: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.508 ms 2025-05-20 02:39:45.328066: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_44Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_44/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_44/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_44/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_44/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 1:18 - loss: 0.6730 - rpn_class_loss: 0.0052 - rpn_bbox_loss: 0.2527 - mrcnn_class_loss: 0.0483 - mrcnn_bbox_loss: 0.2530 - mrcnn_mask_loss: 0.11382025-05-20 02:39:47.053898: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 1:15 - loss: 0.6751 - rpn_class_loss: 0.0050 - rpn_bbox_loss: 0.2511 - mrcnn_class_loss: 0.0473 - mrcnn_bbox_loss: 0.2597 - mrcnn_mask_loss: 0.11192025-05-20 02:39:50.645582: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:39:50.650364: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9989 activity events. 2025-05-20 02:39:51.270675: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_51 2025-05-20 02:39:51.540836: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_51/marlene.trace.json.gz 2025-05-20 02:39:51.828467: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.743 ms 2025-05-20 02:39:51.973169: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_51Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_51/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_51/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_51/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_51/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 1:10 - loss: 0.6617 - rpn_class_loss: 0.0049 - rpn_bbox_loss: 0.2390 - mrcnn_class_loss: 0.0466 - mrcnn_bbox_loss: 0.2617 - mrcnn_mask_loss: 0.10952025-05-20 02:39:52.875514: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 1:08 - loss: 0.6419 - rpn_class_loss: 0.0049 - rpn_bbox_loss: 0.2283 - mrcnn_class_loss: 0.0457 - mrcnn_bbox_loss: 0.2559 - mrcnn_mask_loss: 0.10702025-05-20 02:39:57.526657: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:39:57.530149: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9996 activity events. 2025-05-20 02:39:58.285846: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_57 2025-05-20 02:39:58.453445: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_57/marlene.trace.json.gz 2025-05-20 02:39:58.647617: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.015 ms 2025-05-20 02:39:58.778069: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_57Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_57/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_57/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_57/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_39_57/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 1:05 - loss: 0.6208 - rpn_class_loss: 0.0049 - rpn_bbox_loss: 0.2191 - mrcnn_class_loss: 0.0445 - mrcnn_bbox_loss: 0.2467 - mrcnn_mask_loss: 0.10562025-05-20 02:40:01.274079: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 1:01 - loss: 0.6031 - rpn_class_loss: 0.0049 - rpn_bbox_loss: 0.2107 - mrcnn_class_loss: 0.0443 - mrcnn_bbox_loss: 0.2389 - mrcnn_mask_loss: 0.10432025-05-20 02:40:03.739037: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:40:03.741926: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9988 activity events. 2025-05-20 02:40:04.473743: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_04 2025-05-20 02:40:04.654446: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_04/marlene.trace.json.gz 2025-05-20 02:40:05.027153: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.324 ms 2025-05-20 02:40:05.246288: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_04Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_04/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_04/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_04/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_04/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 58s - loss: 0.6082 - rpn_class_loss: 0.0050 - rpn_bbox_loss: 0.2221 - mrcnn_class_loss: 0.0431 - mrcnn_bbox_loss: 0.2335 - mrcnn_mask_loss: 0.1045 2025-05-20 02:40:08.477736: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 54s - loss: 0.6102 - rpn_class_loss: 0.0052 - rpn_bbox_loss: 0.2307 - mrcnn_class_loss: 0.0416 - mrcnn_bbox_loss: 0.2276 - mrcnn_mask_loss: 0.10522025-05-20 02:40:11.511749: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:40:11.514349: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9996 activity events. 2025-05-20 02:40:12.237297: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_12 2025-05-20 02:40:12.502307: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_12/marlene.trace.json.gz 2025-05-20 02:40:12.732941: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.372 ms 2025-05-20 02:40:12.897594: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_12/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 52s - loss: 0.5996 - rpn_class_loss: 0.0052 - rpn_bbox_loss: 0.2272 - mrcnn_class_loss: 0.0403 - mrcnn_bbox_loss: 0.2222 - mrcnn_mask_loss: 0.10472025-05-20 02:40:16.309243: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 47s - loss: 0.5908 - rpn_class_loss: 0.0053 - rpn_bbox_loss: 0.2249 - mrcnn_class_loss: 0.0393 - mrcnn_bbox_loss: 0.2174 - mrcnn_mask_loss: 0.10392025-05-20 02:40:18.145516: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:40:18.147911: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9988 activity events. 2025-05-20 02:40:18.765373: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_18 2025-05-20 02:40:19.075131: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_18/marlene.trace.json.gz 2025-05-20 02:40:19.397622: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.072 ms 2025-05-20 02:40:19.583070: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_18Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_18/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_18/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_18/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_18/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 45s - loss: 0.5796 - rpn_class_loss: 0.0055 - rpn_bbox_loss: 0.2193 - mrcnn_class_loss: 0.0383 - mrcnn_bbox_loss: 0.2131 - mrcnn_mask_loss: 0.10332025-05-20 02:40:23.939052: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 42s - loss: 0.5684 - rpn_class_loss: 0.0056 - rpn_bbox_loss: 0.2138 - mrcnn_class_loss: 0.0378 - mrcnn_bbox_loss: 0.2084 - mrcnn_mask_loss: 0.10282025-05-20 02:40:27.280860: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:40:27.288225: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9999 activity events. 2025-05-20 02:40:28.379021: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_27 2025-05-20 02:40:28.715180: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_27/marlene.trace.json.gz 2025-05-20 02:40:29.104230: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.3 ms 2025-05-20 02:40:29.353257: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_27Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_27/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_27/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_27/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_27/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 39s - loss: 0.5587 - rpn_class_loss: 0.0055 - rpn_bbox_loss: 0.2105 - mrcnn_class_loss: 0.0369 - mrcnn_bbox_loss: 0.2043 - mrcnn_mask_loss: 0.10142025-05-20 02:40:32.282328: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 36s - loss: 0.5542 - rpn_class_loss: 0.0054 - rpn_bbox_loss: 0.2097 - mrcnn_class_loss: 0.0360 - mrcnn_bbox_loss: 0.2024 - mrcnn_mask_loss: 0.10072025-05-20 02:40:37.581627: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:40:37.590492: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9999 activity events. 2025-05-20 02:40:38.161202: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_37 2025-05-20 02:40:38.477571: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_37/marlene.trace.json.gz 2025-05-20 02:40:38.744656: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.241 ms 2025-05-20 02:40:38.925013: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_37Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_37/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_37/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_37/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_37/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 32s - loss: 0.5467 - rpn_class_loss: 0.0056 - rpn_bbox_loss: 0.2070 - mrcnn_class_loss: 0.0356 - mrcnn_bbox_loss: 0.1989 - mrcnn_mask_loss: 0.09962025-05-20 02:40:42.188735: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 29s - loss: 0.5401 - rpn_class_loss: 0.0057 - rpn_bbox_loss: 0.2049 - mrcnn_class_loss: 0.0353 - mrcnn_bbox_loss: 0.1958 - mrcnn_mask_loss: 0.09842025-05-20 02:40:49.112568: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:40:49.121589: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10046 activity events. 2025-05-20 02:40:49.784112: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_49 2025-05-20 02:40:50.078257: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_49/marlene.trace.json.gz 2025-05-20 02:40:50.392458: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.58 ms 2025-05-20 02:40:50.583558: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_49Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_49/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_49/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_49/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_49/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 26s - loss: 0.5440 - rpn_class_loss: 0.0058 - rpn_bbox_loss: 0.2097 - mrcnn_class_loss: 0.0346 - mrcnn_bbox_loss: 0.1959 - mrcnn_mask_loss: 0.09812025-05-20 02:40:51.920983: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 22s - loss: 0.5443 - rpn_class_loss: 0.0057 - rpn_bbox_loss: 0.2103 - mrcnn_class_loss: 0.0338 - mrcnn_bbox_loss: 0.1967 - mrcnn_mask_loss: 0.09782025-05-20 02:40:58.003168: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:40:58.007258: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9983 callback api events and 10010 activity events. 2025-05-20 02:40:58.560707: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_58 2025-05-20 02:40:58.735337: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_58/marlene.trace.json.gz 2025-05-20 02:40:58.975510: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.285 ms 2025-05-20 02:40:59.151914: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_58Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_58/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_58/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_58/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_40_58/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 18s - loss: 0.5405 - rpn_class_loss: 0.0056 - rpn_bbox_loss: 0.2050 - mrcnn_class_loss: 0.0330 - mrcnn_bbox_loss: 0.1995 - mrcnn_mask_loss: 0.09752025-05-20 02:40:59.618323: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 15s - loss: 0.5354 - rpn_class_loss: 0.0055 - rpn_bbox_loss: 0.1996 - mrcnn_class_loss: 0.0323 - mrcnn_bbox_loss: 0.2008 - mrcnn_mask_loss: 0.09722025-05-20 02:41:04.846164: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:04.849545: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9999 activity events. 2025-05-20 02:41:05.503332: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_05 2025-05-20 02:41:05.747931: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_05/marlene.trace.json.gz 2025-05-20 02:41:05.971299: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.745 ms 2025-05-20 02:41:06.109908: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_05Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_05/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_05/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_05/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_05/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 11s - loss: 0.5279 - rpn_class_loss: 0.0054 - rpn_bbox_loss: 0.1967 - mrcnn_class_loss: 0.0321 - mrcnn_bbox_loss: 0.1973 - mrcnn_mask_loss: 0.09632025-05-20 02:41:06.556276: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 7s - loss: 0.5197 - rpn_class_loss: 0.0053 - rpn_bbox_loss: 0.1941 - mrcnn_class_loss: 0.0316 - mrcnn_bbox_loss: 0.1931 - mrcnn_mask_loss: 0.0955 2025-05-20 02:41:12.032974: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:12.039694: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 10008 activity events. 2025-05-20 02:41:12.760099: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_12 2025-05-20 02:41:12.947137: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_12/marlene.trace.json.gz 2025-05-20 02:41:13.199918: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.061 ms 2025-05-20 02:41:13.372785: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_12/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 3s - loss: 0.5364 - rpn_class_loss: 0.0056 - rpn_bbox_loss: 0.2105 - mrcnn_class_loss: 0.0310 - mrcnn_bbox_loss: 0.1933 - mrcnn_mask_loss: 0.09592025-05-20 02:41:13.842522: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:41:15.700650: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:15.704585: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9990 activity events. 2025-05-20 02:41:16.411834: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_16 2025-05-20 02:41:16.623947: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_16/marlene.trace.json.gz 2025-05-20 02:41:16.883482: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.837 ms 2025-05-20 02:41:17.051573: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_16/marlene.kernel_stats.pb 40/40 [==============================] - 151s 4s/step - loss: 0.5315 - rpn_class_loss: 0.0056 - rpn_bbox_loss: 0.2087 - mrcnn_class_loss: 0.0303 - mrcnn_bbox_loss: 0.1911 - mrcnn_mask_loss: 0.0958 - val_loss: 0.2547 - val_rpn_class_loss: 0.0068 - val_rpn_bbox_loss: 0.0527 - val_mrcnn_class_loss: 0.0167 - val_mrcnn_bbox_loss: 0.1077 - val_mrcnn_mask_loss: 0.0708 Epoch 6/6 1/40 [..............................] - ETA: 13s - loss: 0.9967 - rpn_class_loss: 0.0174 - rpn_bbox_loss: 0.7509 - mrcnn_class_loss: 0.0050 - mrcnn_bbox_loss: 0.1102 - mrcnn_mask_loss: 0.11322025-05-20 02:41:21.739945: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 14s - loss: 0.9905 - rpn_class_loss: 0.0115 - rpn_bbox_loss: 0.5606 - mrcnn_class_loss: 0.0082 - mrcnn_bbox_loss: 0.3241 - mrcnn_mask_loss: 0.08602025-05-20 02:41:22.377384: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:22.380330: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:41:22.951000: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_22 2025-05-20 02:41:23.132207: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_22/marlene.trace.json.gz 2025-05-20 02:41:23.340424: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.937 ms 2025-05-20 02:41:23.454489: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_22Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_22/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_22/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_22/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_22/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 30s - loss: 0.7645 - rpn_class_loss: 0.0081 - rpn_bbox_loss: 0.3813 - mrcnn_class_loss: 0.0093 - mrcnn_bbox_loss: 0.2771 - mrcnn_mask_loss: 0.08862025-05-20 02:41:23.833877: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 44s - loss: 0.6617 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.3127 - mrcnn_class_loss: 0.0144 - mrcnn_bbox_loss: 0.2398 - mrcnn_mask_loss: 0.08722025-05-20 02:41:26.667228: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:26.671078: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9991 activity events. 2025-05-20 02:41:27.121717: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_26 2025-05-20 02:41:27.296528: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_26/marlene.trace.json.gz 2025-05-20 02:41:27.450031: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.035 ms 2025-05-20 02:41:27.553083: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_26/marlene.kernel_stats.pb /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.795911). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.237700). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.220273). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.219241). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.255545). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.275520). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.178381). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.157601). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.142273). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.332320). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.352227). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.107490). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.863360). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.865108). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.938601). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.938611). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.970551). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.789789). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.815224). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.987258). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.183949). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.826904). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.763487). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.766837). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.893387). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.894825). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.938934). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.953242). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.929810). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.868759). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.881389). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.874453). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.832263). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.848019). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.898294). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.902751). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.901220). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.916220). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.909583). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.692171). Check your callbacks. warnings.warn( 5/40 [==>...........................] - ETA: 46s - loss: 0.6710 - rpn_class_loss: 0.0070 - rpn_bbox_loss: 0.2895 - mrcnn_class_loss: 0.0134 - mrcnn_bbox_loss: 0.2795 - mrcnn_mask_loss: 0.08162025-05-20 02:41:27.976892: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 6/40 [===>..........................] - ETA: 50s - loss: 0.7352 - rpn_class_loss: 0.0098 - rpn_bbox_loss: 0.3506 - mrcnn_class_loss: 0.0138 - mrcnn_bbox_loss: 0.2768 - mrcnn_mask_loss: 0.08432025-05-20 02:41:30.470482: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:30.476128: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9991 activity events. 2025-05-20 02:41:31.117682: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_30 2025-05-20 02:41:31.319220: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_30/marlene.trace.json.gz 2025-05-20 02:41:31.583550: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.132 ms 2025-05-20 02:41:31.717156: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_30Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_30/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_30/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_30/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_30/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 1:03 - loss: 0.6665 - rpn_class_loss: 0.0094 - rpn_bbox_loss: 0.3170 - mrcnn_class_loss: 0.0129 - mrcnn_bbox_loss: 0.2443 - mrcnn_mask_loss: 0.08292025-05-20 02:41:34.769001: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 59s - loss: 0.6435 - rpn_class_loss: 0.0092 - rpn_bbox_loss: 0.3003 - mrcnn_class_loss: 0.0130 - mrcnn_bbox_loss: 0.2291 - mrcnn_mask_loss: 0.0920 2025-05-20 02:41:36.870970: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:36.874941: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 9990 activity events. 2025-05-20 02:41:37.659364: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_37 2025-05-20 02:41:37.973430: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_37/marlene.trace.json.gz 2025-05-20 02:41:38.208213: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.305 ms 2025-05-20 02:41:38.388561: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_37Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_37/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_37/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_37/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_37/marlene.kernel_stats.pb /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 9/40 [=====>........................] - ETA: 1:17 - loss: 0.6562 - rpn_class_loss: 0.0103 - rpn_bbox_loss: 0.3166 - mrcnn_class_loss: 0.0138 - mrcnn_bbox_loss: 0.2214 - mrcnn_mask_loss: 0.09412025-05-20 02:41:43.909914: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 1:09 - loss: 0.6346 - rpn_class_loss: 0.0095 - rpn_bbox_loss: 0.3059 - mrcnn_class_loss: 0.0131 - mrcnn_bbox_loss: 0.2119 - mrcnn_mask_loss: 0.09422025-05-20 02:41:44.801786: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:44.806926: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9991 activity events. 2025-05-20 02:41:45.463442: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_45 2025-05-20 02:41:45.718732: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_45/marlene.trace.json.gz 2025-05-20 02:41:45.981955: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.881 ms 2025-05-20 02:41:46.190100: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_45Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_45/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_45/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_45/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_45/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 1:23 - loss: 0.6163 - rpn_class_loss: 0.0097 - rpn_bbox_loss: 0.2899 - mrcnn_class_loss: 0.0126 - mrcnn_bbox_loss: 0.2053 - mrcnn_mask_loss: 0.09872025-05-20 02:41:52.972343: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 1:14 - loss: 0.5901 - rpn_class_loss: 0.0095 - rpn_bbox_loss: 0.2759 - mrcnn_class_loss: 0.0128 - mrcnn_bbox_loss: 0.1963 - mrcnn_mask_loss: 0.09552025-05-20 02:41:53.726232: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:41:53.728900: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:41:54.382673: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_54 2025-05-20 02:41:54.650766: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_54/marlene.trace.json.gz 2025-05-20 02:41:54.869873: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.776 ms 2025-05-20 02:41:55.045487: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_54Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_54/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_54/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_54/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_41_54/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 1:22 - loss: 0.5647 - rpn_class_loss: 0.0090 - rpn_bbox_loss: 0.2611 - mrcnn_class_loss: 0.0126 - mrcnn_bbox_loss: 0.1879 - mrcnn_mask_loss: 0.09412025-05-20 02:42:01.140970: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 1:14 - loss: 0.5515 - rpn_class_loss: 0.0090 - rpn_bbox_loss: 0.2512 - mrcnn_class_loss: 0.0123 - mrcnn_bbox_loss: 0.1839 - mrcnn_mask_loss: 0.09512025-05-20 02:42:02.055557: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:42:02.062557: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9981 activity events. 2025-05-20 02:42:02.601576: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_02 2025-05-20 02:42:02.871272: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_02/marlene.trace.json.gz 2025-05-20 02:42:03.092997: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.889 ms 2025-05-20 02:42:03.801261: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_02Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_02/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_02/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_02/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_02/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 1:16 - loss: 0.5486 - rpn_class_loss: 0.0085 - rpn_bbox_loss: 0.2416 - mrcnn_class_loss: 0.0181 - mrcnn_bbox_loss: 0.1873 - mrcnn_mask_loss: 0.09312025-05-20 02:42:07.121911: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 1:10 - loss: 0.5294 - rpn_class_loss: 0.0084 - rpn_bbox_loss: 0.2324 - mrcnn_class_loss: 0.0183 - mrcnn_bbox_loss: 0.1795 - mrcnn_mask_loss: 0.09082025-05-20 02:42:08.553229: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:42:08.557143: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 9992 activity events. 2025-05-20 02:42:09.315284: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_08 2025-05-20 02:42:09.570608: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_08/marlene.trace.json.gz 2025-05-20 02:42:09.857467: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.547 ms 2025-05-20 02:42:09.987384: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_08Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_08/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_08/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_08/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_08/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 1:07 - loss: 0.5196 - rpn_class_loss: 0.0082 - rpn_bbox_loss: 0.2228 - mrcnn_class_loss: 0.0179 - mrcnn_bbox_loss: 0.1765 - mrcnn_mask_loss: 0.09422025-05-20 02:42:11.389165: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 1:06 - loss: 0.5074 - rpn_class_loss: 0.0081 - rpn_bbox_loss: 0.2145 - mrcnn_class_loss: 0.0174 - mrcnn_bbox_loss: 0.1730 - mrcnn_mask_loss: 0.09442025-05-20 02:42:16.051814: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:42:16.055255: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 10003 activity events. 2025-05-20 02:42:16.486940: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_16 2025-05-20 02:42:16.688451: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_16/marlene.trace.json.gz 2025-05-20 02:42:16.861328: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.333 ms 2025-05-20 02:42:16.989689: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_16/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 1:01 - loss: 0.4877 - rpn_class_loss: 0.0079 - rpn_bbox_loss: 0.2050 - mrcnn_class_loss: 0.0169 - mrcnn_bbox_loss: 0.1651 - mrcnn_mask_loss: 0.09282025-05-20 02:42:17.424092: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 1:01 - loss: 0.4819 - rpn_class_loss: 0.0075 - rpn_bbox_loss: 0.1971 - mrcnn_class_loss: 0.0207 - mrcnn_bbox_loss: 0.1653 - mrcnn_mask_loss: 0.09122025-05-20 02:42:23.013125: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:42:23.016180: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10052 activity events. 2025-05-20 02:42:23.417906: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_23 2025-05-20 02:42:23.647680: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_23/marlene.trace.json.gz 2025-05-20 02:42:23.804085: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.362 ms 2025-05-20 02:42:23.920721: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_23/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 56s - loss: 0.4733 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.1931 - mrcnn_class_loss: 0.0202 - mrcnn_bbox_loss: 0.1617 - mrcnn_mask_loss: 0.0907 2025-05-20 02:42:24.335472: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 54s - loss: 0.4708 - rpn_class_loss: 0.0076 - rpn_bbox_loss: 0.1868 - mrcnn_class_loss: 0.0211 - mrcnn_bbox_loss: 0.1607 - mrcnn_mask_loss: 0.09452025-05-20 02:42:28.265779: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:42:28.267592: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10018 callback api events and 10040 activity events. 2025-05-20 02:42:28.864704: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_28 2025-05-20 02:42:29.165745: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_28/marlene.trace.json.gz 2025-05-20 02:42:29.404953: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.805 ms 2025-05-20 02:42:29.565820: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_28Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_28/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_28/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_28/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_28/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 50s - loss: 0.4768 - rpn_class_loss: 0.0075 - rpn_bbox_loss: 0.1957 - mrcnn_class_loss: 0.0213 - mrcnn_bbox_loss: 0.1584 - mrcnn_mask_loss: 0.09392025-05-20 02:42:30.110745: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 49s - loss: 0.4669 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 0.1912 - mrcnn_class_loss: 0.0206 - mrcnn_bbox_loss: 0.1542 - mrcnn_mask_loss: 0.09312025-05-20 02:42:35.301220: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:42:35.303623: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 10003 activity events. 2025-05-20 02:42:35.952910: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_35 2025-05-20 02:42:36.193053: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_35/marlene.trace.json.gz 2025-05-20 02:42:36.404116: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.894 ms 2025-05-20 02:42:36.532086: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_35Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_35/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_35/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_35/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_35/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 46s - loss: 0.5715 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.2401 - mrcnn_class_loss: 0.0268 - mrcnn_bbox_loss: 0.1700 - mrcnn_mask_loss: 0.12612025-05-20 02:42:39.148872: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 46s - loss: 0.5604 - rpn_class_loss: 0.0087 - rpn_bbox_loss: 0.2346 - mrcnn_class_loss: 0.0261 - mrcnn_bbox_loss: 0.1668 - mrcnn_mask_loss: 0.12422025-05-20 02:42:48.038635: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:42:48.041975: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9980 callback api events and 10040 activity events. 2025-05-20 02:42:48.718889: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_48 2025-05-20 02:42:48.938123: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_48/marlene.trace.json.gz 2025-05-20 02:42:49.188174: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.628 ms 2025-05-20 02:42:49.360044: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_48Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_48/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_48/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_48/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_42_48/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 42s - loss: 0.5551 - rpn_class_loss: 0.0087 - rpn_bbox_loss: 0.2338 - mrcnn_class_loss: 0.0256 - mrcnn_bbox_loss: 0.1640 - mrcnn_mask_loss: 0.12302025-05-20 02:42:49.841721: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 43s - loss: 0.5410 - rpn_class_loss: 0.0085 - rpn_bbox_loss: 0.2264 - mrcnn_class_loss: 0.0248 - mrcnn_bbox_loss: 0.1601 - mrcnn_mask_loss: 0.12132025-05-20 02:43:02.285040: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:43:02.291885: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10018 callback api events and 10090 activity events. 2025-05-20 02:43:02.931547: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_02 2025-05-20 02:43:03.173793: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_02/marlene.trace.json.gz 2025-05-20 02:43:03.456786: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.671 ms 2025-05-20 02:43:03.578448: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_02Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_02/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_02/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_02/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_02/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 38s - loss: 0.5323 - rpn_class_loss: 0.0083 - rpn_bbox_loss: 0.2224 - mrcnn_class_loss: 0.0248 - mrcnn_bbox_loss: 0.1573 - mrcnn_mask_loss: 0.11952025-05-20 02:43:04.020576: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 36s - loss: 0.5948 - rpn_class_loss: 0.0091 - rpn_bbox_loss: 0.2467 - mrcnn_class_loss: 0.0305 - mrcnn_bbox_loss: 0.1680 - mrcnn_mask_loss: 0.14042025-05-20 02:43:12.237115: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:43:12.240115: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10021 callback api events and 10069 activity events. 2025-05-20 02:43:12.764400: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_12 2025-05-20 02:43:13.064955: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_12/marlene.trace.json.gz 2025-05-20 02:43:13.394992: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.131 ms 2025-05-20 02:43:13.579992: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_12/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 32s - loss: 0.6073 - rpn_class_loss: 0.0088 - rpn_bbox_loss: 0.2544 - mrcnn_class_loss: 0.0318 - mrcnn_bbox_loss: 0.1741 - mrcnn_mask_loss: 0.13822025-05-20 02:43:14.062529: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 28s - loss: 0.6715 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.3053 - mrcnn_class_loss: 0.0327 - mrcnn_bbox_loss: 0.1878 - mrcnn_mask_loss: 0.13712025-05-20 02:43:16.155138: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:43:16.160992: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10019 activity events. 2025-05-20 02:43:16.894039: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_16 2025-05-20 02:43:17.299668: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_16/marlene.trace.json.gz 2025-05-20 02:43:17.677488: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.287 ms 2025-05-20 02:43:17.838733: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_16Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_16/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_16/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_16/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_16/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 24s - loss: 0.6612 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.2996 - mrcnn_class_loss: 0.0320 - mrcnn_bbox_loss: 0.1855 - mrcnn_mask_loss: 0.13562025-05-20 02:43:18.798320: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 22s - loss: 0.6693 - rpn_class_loss: 0.0088 - rpn_bbox_loss: 0.3043 - mrcnn_class_loss: 0.0322 - mrcnn_bbox_loss: 0.1880 - mrcnn_mask_loss: 0.13592025-05-20 02:43:27.802296: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:43:27.805045: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 10019 callback api events and 10067 activity events. 2025-05-20 02:43:28.594181: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_28 2025-05-20 02:43:28.850734: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_28/marlene.trace.json.gz 2025-05-20 02:43:29.051957: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.354 ms 2025-05-20 02:43:29.224394: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_28Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_28/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_28/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_28/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_28/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 18s - loss: 0.6572 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.2974 - mrcnn_class_loss: 0.0314 - mrcnn_bbox_loss: 0.1855 - mrcnn_mask_loss: 0.13432025-05-20 02:43:29.719356: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 14s - loss: 0.6508 - rpn_class_loss: 0.0084 - rpn_bbox_loss: 0.2928 - mrcnn_class_loss: 0.0311 - mrcnn_bbox_loss: 0.1860 - mrcnn_mask_loss: 0.13252025-05-20 02:43:35.806533: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:43:35.812600: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 10017 activity events. 2025-05-20 02:43:36.726986: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_36 2025-05-20 02:43:37.144242: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_36/marlene.trace.json.gz 2025-05-20 02:43:37.508906: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.707 ms 2025-05-20 02:43:37.725987: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_36Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_36/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_36/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_36/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_36/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 11s - loss: 0.6567 - rpn_class_loss: 0.0082 - rpn_bbox_loss: 0.2964 - mrcnn_class_loss: 0.0318 - mrcnn_bbox_loss: 0.1893 - mrcnn_mask_loss: 0.13102025-05-20 02:43:40.563474: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 7s - loss: 0.6492 - rpn_class_loss: 0.0082 - rpn_bbox_loss: 0.2926 - mrcnn_class_loss: 0.0314 - mrcnn_bbox_loss: 0.1872 - mrcnn_mask_loss: 0.1298 2025-05-20 02:43:45.614929: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:43:45.618274: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 10005 activity events. 2025-05-20 02:43:46.390190: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_45 2025-05-20 02:43:46.719088: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_45/marlene.trace.json.gz 2025-05-20 02:43:47.162534: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.521 ms 2025-05-20 02:43:47.528051: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_45Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_45/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_45/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_45/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_45/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 3s - loss: 0.6732 - rpn_class_loss: 0.0080 - rpn_bbox_loss: 0.3036 - mrcnn_class_loss: 0.0320 - mrcnn_bbox_loss: 0.2008 - mrcnn_mask_loss: 0.12892025-05-20 02:43:51.357057: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-20 02:43:54.406276: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-20 02:43:54.411798: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9981 callback api events and 10007 activity events. 2025-05-20 02:43:55.063779: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_54 2025-05-20 02:43:55.362239: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_54/marlene.trace.json.gz 2025-05-20 02:43:55.683567: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.14 ms 2025-05-20 02:43:55.856949: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_54Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_54/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_54/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_54/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_200525_2/learn_mask_pancarte_200525_220250520T0226/train/plugins/profile/2025_05_20_02_43_54/marlene.kernel_stats.pb 40/40 [==============================] - 158s 4s/step - loss: 0.6655 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 0.2992 - mrcnn_class_loss: 0.0320 - mrcnn_bbox_loss: 0.1995 - mrcnn_mask_loss: 0.1270 - val_loss: 0.3853 - val_rpn_class_loss: 0.0052 - val_rpn_bbox_loss: 0.1454 - val_mrcnn_class_loss: 0.0087 - val_mrcnn_bbox_loss: 0.1491 - val_mrcnn_mask_loss: 0.0768 /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.623681). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.624654). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.635753). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.796381). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.817334). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.668538). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.611800). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.609617). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.633805). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.686933). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.880228). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.916485). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.969878). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.064984). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.061678). Check your callbacks. warnings.warn( model_param file didn't exist model didn't exist , save the new model in s3 local folder to save in s3 : /data/models_weight/learn_mask_pancarte_200525_2 update : 1747701844.4432604 done ! 1747701849.4997485 {'files': [{'name': 'mask_model.h5', 'size': 255878432, 'last_modified': '2025-05-20T00:44:04.480550', 'hash': 'e307465d6b58f4eaac380253208a3ebb'}], 'directories': []} create thcl : begin to create parametres in sql : time : 2025-05-20 02:44:09 insert information in MTRPhoto.photo_desc_type_params: INSERT INTO MTRPhoto.photo_desc_type_params (name, svm_number_of_descriptors, size_desc, caffemodel_name, layer_name, scale_desc, nb_discretization, crop_stripes, nb_bit, shift_desc, default_storage_type, created_at, updated_at) VALUES ('learn_mask_pancarte_200525_2', '16384', '25088', 'learn_mask_pancarte_200525_2', 'pool5', '10', '256', '0', '8', '-1000', '3', '2025-05-20 02:44:09', '2025-05-20 02:44:09'); insert le photo_hashtag_type: INSERT INTO MTRBack.photo_hashtag_type (id, type) VALUES ('5008', 'learn_mask_pancarte_200525_2'); insert information in MTRDatou.classification_theme: INSERT INTO MTRDatou.classification_theme (mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification) VALUES ('31', 'learn_mask_pancarte_200525_2', '0', '0,0,0', 'background,badge,pancarte', '0,0,0', '5008', '6104', '10000', '1000', 0, 0, '2025-05-20 02:44:09', '2025-05-20 02:44:09', 'mask_rcnn'); photo_hashtag_type id: 5008 photo_desc_type id: 6104 manual tagging problem id: 3910 insert informations in datou : thcl : 3910 insert model in mtr_datou : insert step in MTRPhoto.mtr_datou_step datou_id : 4937 (step id 14085) done! UPDATE MTRDatou.mtr_datou_step SET param_json = '{"photo_hashtag_type":5008}' WHERE mtd_id = '4937'AND type = 1; # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! execution d'update_param_json param_json_actuel: ((None,),) self.json_param: {} new_param_json: {"pht": 5007} UPDATE MTRDatou.mtr_datou_step SET param_json='{"pht": 5007}' WHERE id=14085; fin d'execution d'update_param_json 904.21user 3076.59system 20:02.84elapsed 330%CPU (0avgtext+0avgdata 4678760maxresident)k 4711952inputs+6495352outputs (172143major+95877921minor)pagefaults 0swaps