python /home/admin/mtr/script_for_cron.py -j coverage -m 9 -a '' -s coverage -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded root_folder /data_4/data_log/job/2025/November/20112025/coverage/ git_velours : /home/admin/workarea/git/Velours/ out_folder_name htmlcov output_folder /data_4/data_log/job/2025/November/20112025/coverage/htmlcov new path : /data_4/data_log/job/2025/November/20112025/coverage/ command : coverage3 run /home/admin/workarea/git/Velours/python/tests/python_tests.py --short_python3 `cat ~/.fotonower_pass/bdd.py.pass` cat: /home/admin/.fotonower_pass/bdd.py.pass: Aucun fichier ou dossier de ce type import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data python version used : 3 #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10998 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.12553954124450684 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu Nov 20 05:20:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10998 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-11-20 05:20:31.846849: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-11-20 05:20:31.854690: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2025-11-20 05:20:31.856337: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f61f0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-11-20 05:20:31.856391: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-11-20 05:20:31.859165: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-11-20 05:20:32.146489: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11954820 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-11-20 05:20:32.146550: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-11-20 05:20:32.147542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 05:20:32.147850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 05:20:32.149923: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 05:20:32.152048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 05:20:32.152363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 05:20:32.154644: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 05:20:32.155712: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 05:20:32.160241: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 05:20:32.161682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 05:20:32.161746: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 05:20:32.162527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-11-20 05:20:32.162545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-11-20 05:20:32.162554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-11-20 05:20:32.163922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10193 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-11-20 05:20:32.932090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 05:20:32.932202: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 05:20:32.932223: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 05:20:32.932246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 05:20:32.932267: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 05:20:32.932289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 05:20:32.932310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 05:20:32.932333: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 05:20:32.933702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 05:20:32.935270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 05:20:32.935345: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 05:20:32.935363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 05:20:32.935378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 05:20:32.935392: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 05:20:32.935407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 05:20:32.935422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 05:20:32.935436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 05:20:32.936686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 05:20:32.936726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-11-20 05:20:32.936735: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-11-20 05:20:32.936743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-11-20 05:20:32.938005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10193 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-11-20 05:20:41.616500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 05:20:41.783636: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 05:20:43.957372: E tensorflow/stream_executor/cuda/cuda_driver.cc:910] failed to synchronize the stop event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-11-20 05:20:43.957492: E tensorflow/stream_executor/gpu/gpu_timer.cc:55] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-11-20 05:20:43.957535: E tensorflow/stream_executor/gpu/gpu_timer.cc:60] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-11-20 05:20:43.957593: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-11-20 05:20:43.957609: E tensorflow/stream_executor/stream.cc:5485] Internal: Failed to enqueue async memset operation: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-11-20 05:20:43.957633: W tensorflow/core/kernels/gpu_utils.cc:69] Failed to check cudnn convolutions for out-of-bounds reads and writes with an error message: 'Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered'; skipping this check. This only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-11-20 05:20:43.957644: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-11-20 05:20:43.957817: I tensorflow/stream_executor/stream.cc:4963] [stream=0x123be3a0,impl=0x123bd390] did not memzero GPU location; source: 0x7f60797f8020 2025-11-20 05:20:43.961805: F ./tensorflow/core/kernels/reduction_gpu_kernels.cu.h:731] Non-OK-status: GpuLaunchKernel(RowReduceKernel, num_blocks, threads_per_block, 0, cu_stream, in, out, num_rows, num_cols, op, init) status: Internal: an illegal memory access was encountered max_time_sub_proc : 3600 Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3366 error , can't release the memory or there are other process who occupe the free memory ERROR test release memory FAILED ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False Catched exception ! Connect or reconnect ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.31614208221435547 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu Nov 20 06:20:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3366 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-11-20 06:20:36.980344: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-11-20 06:20:37.010487: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2025-11-20 06:20:37.012597: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f61f0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-11-20 06:20:37.012640: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-11-20 06:20:37.016187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-11-20 06:20:37.265731: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11957f80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-11-20 06:20:37.265808: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-11-20 06:20:37.267240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 06:20:37.267695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 06:20:37.270811: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 06:20:37.288823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 06:20:37.289340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 06:20:37.311356: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 06:20:37.315888: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 06:20:37.334623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 06:20:37.335818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 06:20:37.335886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 06:20:37.336529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-11-20 06:20:37.336547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-11-20 06:20:37.336557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-11-20 06:20:37.337666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2914 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-11-20 06:20:38.215351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 06:20:38.215443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 06:20:38.215464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 06:20:38.215483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 06:20:38.215500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 06:20:38.215517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 06:20:38.215533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 06:20:38.215551: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 06:20:38.216358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 06:20:38.217337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 06:20:38.217375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 06:20:38.217393: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 06:20:38.217410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 06:20:38.217426: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 06:20:38.217442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 06:20:38.217458: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 06:20:38.217474: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 06:20:38.218294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 06:20:38.218327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-11-20 06:20:38.218335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-11-20 06:20:38.218342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-11-20 06:20:38.219222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2914 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-11-20 06:20:46.639723: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 06:20:46.819152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 06:20:48.178120: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.178742: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.11G (2268581120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.178766: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.179402: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.179417: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.186553: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.186576: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.187209: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.187225: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.193744: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.193763: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.194359: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.194377: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.225136: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.225166: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.225776: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.225791: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.231655: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.231676: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.232311: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.232329: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:20:48.266426: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.267088: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.268976: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.269565: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.313728: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.314399: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.316709: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.317371: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.325393: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.326036: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.330974: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.331611: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.344317: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.344919: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.346696: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.347334: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.353810: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.354466: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.356367: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.356990: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.363443: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.364085: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.365663: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.366256: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.393075: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.393734: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.394385: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.395025: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.398515: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.399155: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.414521: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.415171: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.415804: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.416436: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.428709: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.429358: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.429993: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.430643: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.434979: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.435610: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.440256: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.440892: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.452993: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.453625: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.457805: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.458454: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.459090: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.459722: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.480396: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.480989: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.481588: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.482173: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.482819: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.483464: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.497862: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.498474: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.524107: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.524183: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-11-20 06:20:48.525240: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.526288: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.533915: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.534979: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.536025: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.537043: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.545758: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.546863: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.563229: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.564305: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.565346: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.566375: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.571138: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.572191: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.573212: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.574231: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.575911: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.586085: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.587138: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.597580: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.598746: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.599625: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.600487: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.601324: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:20:48.602136: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (720, 1280, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 1280.00000 nb d'objets trouves : 4 Detection mask done ! Trying to reset tf kernel 3317584 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2173 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3366 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0004086494445800781 nb_pixel_total : 16902 time to create 1 rle with old method : 0.04155611991882324 length of segment : 107 time for calcul the mask position with numpy : 0.01696300506591797 nb_pixel_total : 480749 time to create 1 rle with new method : 0.02967524528503418 length of segment : 632 time for calcul the mask position with numpy : 0.0004146099090576172 nb_pixel_total : 36583 time to create 1 rle with old method : 0.07801556587219238 length of segment : 132 time for calcul the mask position with numpy : 0.00010633468627929688 nb_pixel_total : 4794 time to create 1 rle with old method : 0.010501861572265625 length of segment : 51 time spent for convertir_results : 1.1579794883728027 time spend for datou_step_exec : 21.815414905548096 time spend to save output : 4.935264587402344e-05 total time spend for step 1 : 21.81546425819397 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 447 chid ids of type : 445 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.020499229431152344 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'917855882': [[(917855882, 492601069, 445, 1092, 1280, 0, 108, 0.9988362, [(1205, 1, 58), (1165, 2, 105), (1159, 3, 113), (1149, 4, 124), (1113, 5, 161), (1100, 6, 174), (1097, 7, 177), (1095, 8, 179), (1095, 9, 179), (1095, 10, 179), (1095, 11, 179), (1095, 12, 179), (1095, 13, 179), (1095, 14, 178), (1095, 15, 178), (1095, 16, 178), (1095, 17, 178), (1095, 18, 177), (1095, 19, 177), (1095, 20, 177), (1095, 21, 177), (1095, 22, 177), (1095, 23, 178), (1095, 24, 178), (1095, 25, 178), (1095, 26, 179), (1095, 27, 179), (1095, 28, 180), (1095, 29, 181), (1095, 30, 182), (1095, 31, 183), (1095, 32, 183), (1095, 33, 184), (1095, 34, 184), (1096, 35, 183), (1096, 36, 183), (1096, 37, 184), (1097, 38, 183), (1097, 39, 183), (1097, 40, 183), (1098, 41, 182), (1098, 42, 182), (1098, 43, 182), (1099, 44, 181), (1099, 45, 181), (1099, 46, 181), (1100, 47, 180), (1100, 48, 180), (1101, 49, 179), (1101, 50, 179), (1102, 51, 178), (1102, 52, 178), (1103, 53, 177), (1103, 54, 177), (1104, 55, 176), (1104, 56, 176), (1104, 57, 176), (1104, 58, 176), (1105, 59, 175), (1105, 60, 175), (1105, 61, 175), (1105, 62, 175), (1105, 63, 175), (1106, 64, 174), (1106, 65, 174), (1106, 66, 174), (1106, 67, 174), (1106, 68, 174), (1106, 69, 174), (1106, 70, 174), (1106, 71, 174), (1106, 72, 174), (1106, 73, 174), (1107, 74, 173), (1107, 75, 173), (1107, 76, 173), (1107, 77, 173), (1107, 78, 173), (1107, 79, 173), (1108, 80, 172), (1108, 81, 172), (1109, 82, 171), (1110, 83, 170), (1110, 84, 170), (1111, 85, 169), (1112, 86, 168), (1113, 87, 166), (1114, 88, 165), (1115, 89, 164), (1117, 90, 162), (1120, 91, 159), (1138, 92, 141), (1146, 93, 133), (1154, 94, 125), (1167, 95, 112), (1177, 96, 102), (1183, 97, 95), (1185, 98, 93), (1187, 99, 90), (1188, 100, 55), (1264, 100, 12), (1190, 101, 50), (1191, 102, 46), (1194, 103, 40), (1197, 104, 34), (1202, 105, 25), (1207, 106, 16)], ['1222,106,1207,106,1206,105,1197,104,1191,102,1182,96,1176,95,1167,95,1166,94,1154,94,1153,93,1146,93,1145,92,1137,91,1120,91,1115,89,1110,84,1107,79,1106,73,1106,64,1104,55,1099,46,1095,34,1095,8,1100,6,1112,6,1113,5,1148,5,1149,4,1158,4,1165,2,1204,2,1205,1,1262,1,1269,2,1273,5,1273,13,1271,18,1271,22,1273,27,1277,31,1279,37,1279,86,1278,87,1278,96,1275,100,1264,100,1263,99,1243,99,1230,104']), (917855882, 492601069, 445, 52, 1128, 16, 668, 0.9977464, [(711, 22, 21), (925, 22, 47), (608, 23, 146), (894, 23, 103), (598, 24, 234), (850, 24, 158), (590, 25, 427), (582, 26, 444), (575, 27, 458), (569, 28, 466), (565, 29, 472), (560, 30, 480), (556, 31, 486), (550, 32, 495), (544, 33, 503), (538, 34, 512), (532, 35, 520), (527, 36, 527), (523, 37, 534), (518, 38, 541), (514, 39, 548), (510, 40, 554), (506, 41, 561), (503, 42, 566), (499, 43, 572), (496, 44, 577), (493, 45, 582), (491, 46, 585), (489, 47, 589), (487, 48, 592), (485, 49, 595), (483, 50, 598), (482, 51, 600), (481, 52, 602), (480, 53, 603), (479, 54, 605), (478, 55, 606), (477, 56, 607), (475, 57, 610), (474, 58, 611), (473, 59, 613), (472, 60, 614), (470, 61, 616), (469, 62, 618), (468, 63, 619), (466, 64, 621), (465, 65, 623), (464, 66, 624), (462, 67, 626), (461, 68, 628), (459, 69, 630), (458, 70, 631), (456, 71, 633), (455, 72, 635), (453, 73, 637), (452, 74, 638), (451, 75, 639), (450, 76, 640), (448, 77, 642), (447, 78, 643), (446, 79, 644), (445, 80, 645), (444, 81, 646), (442, 82, 648), (441, 83, 649), (440, 84, 650), (439, 85, 651), (438, 86, 652), (437, 87, 653), (436, 88, 654), (435, 89, 655), (434, 90, 656), (433, 91, 657), (432, 92, 658), (431, 93, 659), (430, 94, 660), (429, 95, 661), (428, 96, 662), (427, 97, 663), (425, 98, 665), (423, 99, 667), (421, 100, 669), (419, 101, 671), (417, 102, 673), (413, 103, 677), (410, 104, 680), (406, 105, 684), (401, 106, 689), (397, 107, 693), (392, 108, 698), (387, 109, 703), (382, 110, 708), (377, 111, 713), (373, 112, 717), (369, 113, 721), (365, 114, 725), (362, 115, 728), (358, 116, 732), (356, 117, 734), (353, 118, 737), (351, 119, 739), (349, 120, 741), (346, 121, 744), (344, 122, 746), (341, 123, 749), (338, 124, 752), (335, 125, 755), (331, 126, 759), (327, 127, 763), (323, 128, 767), (319, 129, 770), (314, 130, 775), (308, 131, 781), (303, 132, 786), (294, 133, 795), (287, 134, 802), (279, 135, 810), (273, 136, 816), (267, 137, 822), (262, 138, 827), (258, 139, 831), (255, 140, 834), (252, 141, 837), (250, 142, 839), (247, 143, 842), (245, 144, 844), (242, 145, 847), (240, 146, 849), (237, 147, 852), (234, 148, 855), (230, 149, 859), (226, 150, 863), (220, 151, 869), (213, 152, 876), (207, 153, 882), (200, 154, 889), (193, 155, 896), (187, 156, 902), (183, 157, 906), (181, 158, 908), (178, 159, 911), (176, 160, 913), (174, 161, 915), (172, 162, 917), (170, 163, 919), (168, 164, 921), (167, 165, 922), (165, 166, 924), (164, 167, 925), (162, 168, 927), (161, 169, 928), (159, 170, 930), (157, 171, 932), (155, 172, 934), (153, 173, 935), (151, 174, 937), (148, 175, 940), (146, 176, 942), (144, 177, 944), (142, 178, 946), (140, 179, 948), (139, 180, 949), (137, 181, 951), (136, 182, 952), (134, 183, 954), (133, 184, 955), (132, 185, 956), (131, 186, 957), (130, 187, 958), (129, 188, 959), (128, 189, 960), (127, 190, 960), (126, 191, 961), (126, 192, 961), (125, 193, 962), (124, 194, 963), (123, 195, 964), (122, 196, 965), (122, 197, 965), (121, 198, 966), (120, 199, 967), (119, 200, 968), (118, 201, 969), (117, 202, 970), (116, 203, 971), (114, 204, 973), (113, 205, 973), (112, 206, 974), (111, 207, 975), (109, 208, 977), (108, 209, 978), (107, 210, 979), (106, 211, 980), (105, 212, 981), (104, 213, 982), (103, 214, 983), (102, 215, 984), (101, 216, 985), (101, 217, 984), (100, 218, 985), (100, 219, 985), (99, 220, 986), (98, 221, 987), (98, 222, 987), (97, 223, 988), (97, 224, 987), (96, 225, 988), (96, 226, 988), (95, 227, 989), (95, 228, 989), (94, 229, 990), (94, 230, 990), (94, 231, 990), (93, 232, 990), (93, 233, 990), (92, 234, 991), (92, 235, 991), (92, 236, 991), (91, 237, 992), (91, 238, 991), (91, 239, 991), (91, 240, 991), (91, 241, 990), (90, 242, 991), (90, 243, 990), (90, 244, 990), (90, 245, 989), (90, 246, 989), (89, 247, 990), (89, 248, 989), (89, 249, 989), (89, 250, 988), (89, 251, 988), (88, 252, 988), (88, 253, 988), (88, 254, 987), (88, 255, 986), (88, 256, 986), (87, 257, 986), (87, 258, 986), (87, 259, 985), (87, 260, 984), (87, 261, 983), (86, 262, 983), (86, 263, 983), (86, 264, 982), (86, 265, 981), (85, 266, 981), (85, 267, 980), (85, 268, 980), (84, 269, 980), (84, 270, 979), (84, 271, 979), (84, 272, 978), (83, 273, 979), (83, 274, 978), (83, 275, 977), (82, 276, 978), (82, 277, 977), (82, 278, 977), (81, 279, 977), (81, 280, 977), (81, 281, 977), (80, 282, 977), (80, 283, 977), (80, 284, 976), (79, 285, 977), (79, 286, 976), (79, 287, 976), (78, 288, 976), (78, 289, 976), (78, 290, 975), (77, 291, 976), (77, 292, 975), (77, 293, 975), (76, 294, 975), (76, 295, 975), (76, 296, 974), (75, 297, 975), (75, 298, 974), (74, 299, 975), (74, 300, 974), (74, 301, 974), (73, 302, 974), (73, 303, 974), (72, 304, 974), (72, 305, 974), (71, 306, 974), (71, 307, 973), (71, 308, 972), (70, 309, 972), (70, 310, 971), (70, 311, 970), (70, 312, 968), (69, 313, 968), (69, 314, 966), (69, 315, 964), (69, 316, 962), (68, 317, 961), (68, 318, 959), (68, 319, 958), (68, 320, 956), (67, 321, 955), (67, 322, 954), (67, 323, 952), (67, 324, 951), (66, 325, 951), (66, 326, 950), (66, 327, 948), (66, 328, 947), (65, 329, 947), (65, 330, 946), (65, 331, 946), (65, 332, 945), (65, 333, 944), (65, 334, 942), (65, 335, 941), (65, 336, 940), (65, 337, 939), (65, 338, 938), (64, 339, 937), (64, 340, 936), (64, 341, 934), (64, 342, 932), (64, 343, 930), (64, 344, 928), (64, 345, 926), (64, 346, 925), (64, 347, 923), (64, 348, 922), (64, 349, 920), (64, 350, 919), (63, 351, 919), (63, 352, 918), (63, 353, 917), (63, 354, 916), (63, 355, 915), (63, 356, 914), (63, 357, 912), (63, 358, 911), (63, 359, 910), (63, 360, 909), (63, 361, 908), (63, 362, 906), (63, 363, 905), (63, 364, 904), (63, 365, 902), (63, 366, 901), (63, 367, 899), (63, 368, 898), (62, 369, 897), (62, 370, 895), (62, 371, 893), (62, 372, 891), (62, 373, 890), (62, 374, 888), (62, 375, 887), (62, 376, 886), (62, 377, 885), (62, 378, 884), (62, 379, 883), (63, 380, 880), (63, 381, 879), (63, 382, 878), (63, 383, 877), (63, 384, 876), (63, 385, 875), (63, 386, 874), (63, 387, 873), (63, 388, 872), (64, 389, 870), (64, 390, 869), (64, 391, 868), (64, 392, 867), (64, 393, 865), (64, 394, 864), (64, 395, 863), (64, 396, 862), (65, 397, 860), (65, 398, 859), (65, 399, 858), (65, 400, 857), (65, 401, 856), (65, 402, 854), (65, 403, 853), (65, 404, 851), (65, 405, 850), (65, 406, 848), (66, 407, 846), (66, 408, 844), (66, 409, 843), (66, 410, 842), (66, 411, 841), (66, 412, 840), (66, 413, 838), (66, 414, 837), (66, 415, 836), (66, 416, 835), (66, 417, 835), (66, 418, 834), (66, 419, 833), (66, 420, 832), (67, 421, 830), (67, 422, 829), (67, 423, 829), (67, 424, 828), (67, 425, 827), (67, 426, 826), (67, 427, 825), (67, 428, 824), (67, 429, 823), (68, 430, 820), (68, 431, 819), (68, 432, 818), (68, 433, 816), (68, 434, 815), (68, 435, 813), (68, 436, 811), (69, 437, 809), (69, 438, 807), (69, 439, 806), (69, 440, 804), (69, 441, 803), (69, 442, 802), (69, 443, 800), (70, 444, 798), (70, 445, 797), (70, 446, 796), (70, 447, 796), (71, 448, 794), (71, 449, 794), (72, 450, 792), (72, 451, 792), (73, 452, 790), (73, 453, 789), (74, 454, 788), (74, 455, 787), (75, 456, 786), (75, 457, 785), (76, 458, 784), (76, 459, 783), (77, 460, 782), (77, 461, 781), (77, 462, 781), (78, 463, 779), (78, 464, 779), (79, 465, 777), (79, 466, 777), (79, 467, 776), (80, 468, 775), (80, 469, 774), (80, 470, 774), (81, 471, 772), (81, 472, 771), (82, 473, 770), (82, 474, 769), (83, 475, 767), (83, 476, 766), (83, 477, 766), (84, 478, 764), (84, 479, 763), (85, 480, 761), (85, 481, 760), (85, 482, 759), (86, 483, 757), (86, 484, 755), (87, 485, 753), (87, 486, 752), (87, 487, 751), (88, 488, 748), (88, 489, 747), (88, 490, 746), (89, 491, 744), (89, 492, 743), (90, 493, 741), (90, 494, 741), (91, 495, 739), (91, 496, 738), (92, 497, 737), (93, 498, 735), (94, 499, 733), (94, 500, 733), (95, 501, 731), (96, 502, 729), (97, 503, 728), (98, 504, 726), (99, 505, 724), (99, 506, 724), (100, 507, 722), (101, 508, 721), (102, 509, 719), (104, 510, 717), (105, 511, 715), (106, 512, 714), (107, 513, 712), (108, 514, 711), (110, 515, 708), (111, 516, 707), (113, 517, 704), (114, 518, 703), (115, 519, 701), (117, 520, 698), (118, 521, 697), (119, 522, 695), (121, 523, 693), (122, 524, 691), (124, 525, 689), (125, 526, 687), (126, 527, 685), (128, 528, 683), (129, 529, 681), (131, 530, 678), (132, 531, 676), (134, 532, 673), (135, 533, 672), (137, 534, 669), (138, 535, 667), (140, 536, 664), (141, 537, 662), (143, 538, 659), (144, 539, 657), (146, 540, 654), (148, 541, 651), (149, 542, 649), (151, 543, 645), (153, 544, 642), (154, 545, 640), (156, 546, 638), (158, 547, 635), (159, 548, 633), (161, 549, 630), (162, 550, 628), (164, 551, 626), (165, 552, 624), (167, 553, 621), (168, 554, 619), (170, 555, 617), (171, 556, 615), (173, 557, 613), (174, 558, 611), (176, 559, 608), (177, 560, 607), (178, 561, 605), (180, 562, 603), (181, 563, 601), (183, 564, 599), (184, 565, 598), (186, 566, 595), (189, 567, 592), (192, 568, 589), (195, 569, 585), (198, 570, 582), (201, 571, 579), (203, 572, 576), (206, 573, 573), (209, 574, 569), (212, 575, 566), (215, 576, 563), (218, 577, 559), (221, 578, 556), (223, 579, 553), (226, 580, 550), (228, 581, 547), (230, 582, 545), (232, 583, 542), (234, 584, 540), (235, 585, 539), (237, 586, 536), (238, 587, 534), (240, 588, 531), (242, 589, 528), (243, 590, 526), (245, 591, 523), (247, 592, 520), (249, 593, 516), (251, 594, 513), (253, 595, 510), (256, 596, 505), (258, 597, 501), (261, 598, 497), (263, 599, 493), (267, 600, 488), (271, 601, 482), (274, 602, 478), (278, 603, 473), (281, 604, 468), (284, 605, 464), (287, 606, 460), (290, 607, 456), (292, 608, 453), (295, 609, 449), (298, 610, 445), (300, 611, 442), (303, 612, 438), (305, 613, 434), (308, 614, 430), (310, 615, 427), (312, 616, 423), (315, 617, 418), (317, 618, 415), (320, 619, 410), (322, 620, 406), (325, 621, 401), (328, 622, 395), (330, 623, 390), (333, 624, 384), (335, 625, 379), (338, 626, 374), (341, 627, 369), (345, 628, 362), (349, 629, 356), (353, 630, 350), (357, 631, 344), (360, 632, 340), (364, 633, 334), (368, 634, 327), (373, 635, 320), (378, 636, 313), (383, 637, 305), (389, 638, 295), (395, 639, 282), (401, 640, 270), (408, 641, 256), (416, 642, 240), (432, 643, 216), (448, 644, 193), (465, 645, 169), (480, 646, 148), (495, 647, 126), (511, 648, 104), (526, 649, 82), (565, 650, 9)], ['526,649,416,642,341,627,243,590,220,577,186,566,102,509,91,496,70,447,62,379,65,329,86,265,91,237,101,216,134,183,187,156,225,151,252,141,343,123,358,116,405,106,426,98,442,82,493,45,527,36,608,23,754,24,893,24,925,22,996,23,1032,27,1066,41,1082,52,1089,72,1088,172,1082,237,1045,305,1019,322,1002,338,950,373,885,432,865,446,851,473,822,505,810,528,786,554,773,585,714,624,690,636,607,649']), (917855882, 492601069, 445, 0, 438, 0, 116, 0.99194044, [(127, 1, 140), (94, 2, 206), (59, 3, 273), (338, 3, 59), (22, 4, 380), (19, 5, 386), (16, 6, 391), (15, 7, 393), (14, 8, 395), (14, 9, 396), (13, 10, 398), (12, 11, 399), (12, 12, 399), (11, 13, 401), (10, 14, 402), (11, 15, 402), (11, 16, 403), (12, 17, 403), (12, 18, 404), (12, 19, 405), (12, 20, 405), (12, 21, 405), (12, 22, 406), (12, 23, 406), (12, 24, 406), (12, 25, 407), (12, 26, 407), (12, 27, 407), (12, 28, 408), (12, 29, 408), (12, 30, 408), (12, 31, 408), (12, 32, 408), (12, 33, 408), (12, 34, 409), (12, 35, 409), (12, 36, 409), (12, 37, 409), (12, 38, 409), (12, 39, 409), (12, 40, 409), (12, 41, 409), (12, 42, 409), (12, 43, 410), (12, 44, 410), (12, 45, 409), (12, 46, 409), (12, 47, 409), (12, 48, 409), (12, 49, 409), (12, 50, 409), (12, 51, 409), (12, 52, 408), (12, 53, 408), (12, 54, 407), (12, 55, 406), (12, 56, 405), (12, 57, 404), (12, 58, 402), (11, 59, 402), (11, 60, 401), (11, 61, 401), (11, 62, 400), (11, 63, 399), (11, 64, 399), (11, 65, 398), (11, 66, 397), (11, 67, 396), (11, 68, 395), (11, 69, 395), (11, 70, 394), (11, 71, 394), (11, 72, 394), (11, 73, 393), (11, 74, 393), (11, 75, 393), (11, 76, 393), (11, 77, 393), (11, 78, 393), (11, 79, 392), (11, 80, 392), (10, 81, 393), (10, 82, 394), (10, 83, 394), (9, 84, 396), (9, 85, 263), (284, 85, 121), (9, 86, 75), (97, 86, 29), (141, 86, 120), (292, 86, 112), (9, 87, 71), (152, 87, 104), (294, 87, 110), (8, 88, 67), (160, 88, 92), (295, 88, 108), (8, 89, 63), (175, 89, 73), (296, 89, 107), (7, 90, 61), (204, 90, 41), (297, 90, 105), (7, 91, 57), (298, 91, 104), (7, 92, 53), (299, 92, 103), (6, 93, 50), (300, 93, 101), (7, 94, 46), (303, 94, 96), (7, 95, 44), (305, 95, 93), (7, 96, 42), (308, 96, 88), (7, 97, 40), (310, 97, 85), (7, 98, 38), (312, 98, 82), (8, 99, 34), (314, 99, 79), (8, 100, 32), (316, 100, 75), (8, 101, 29), (319, 101, 71), (13, 102, 18), (324, 102, 63), (20, 103, 6), (331, 103, 51), (337, 104, 37), (344, 105, 22), (351, 106, 2)], ['344,105,330,102,319,101,307,95,300,93,283,84,261,85,244,90,204,90,203,89,175,89,160,88,140,85,125,86,97,86,84,85,67,90,56,92,36,101,25,103,8,101,6,93,11,80,11,59,12,58,12,17,10,14,16,6,22,4,58,4,59,3,93,3,94,2,126,2,127,1,266,1,267,2,331,3,396,3,406,6,416,19,420,34,420,51,411,60,405,68,402,81,404,85,401,92,386,102,365,105']), (917855882, 492601069, 445, 390, 550, 0, 54, 0.939199, [(414, 0, 7), (441, 0, 60), (508, 0, 28), (402, 1, 142), (401, 2, 146), (402, 3, 145), (404, 4, 143), (406, 5, 140), (408, 6, 137), (410, 7, 134), (411, 8, 132), (412, 9, 130), (413, 10, 127), (414, 11, 125), (415, 12, 123), (415, 13, 122), (416, 14, 120), (417, 15, 117), (417, 16, 116), (418, 17, 114), (418, 18, 113), (418, 19, 111), (418, 20, 109), (419, 21, 107), (419, 22, 105), (419, 23, 103), (419, 24, 102), (419, 25, 100), (420, 26, 97), (420, 27, 95), (420, 28, 94), (421, 29, 91), (421, 30, 90), (422, 31, 88), (422, 32, 88), (422, 33, 87), (423, 34, 84), (423, 35, 82), (423, 36, 81), (424, 37, 79), (424, 38, 77), (424, 39, 75), (424, 40, 73), (424, 41, 71), (425, 42, 67), (425, 43, 66), (426, 44, 62), (426, 45, 6), (433, 45, 52), (443, 46, 30), (450, 47, 1)], ['450,47,449,46,443,46,442,45,426,45,424,41,424,37,423,36,422,31,419,25,419,21,418,20,418,17,417,15,409,6,402,3,402,1,413,1,414,0,420,0,421,1,440,1,441,0,500,0,501,1,507,1,508,0,535,0,536,1,543,1,546,2,546,4,542,8,530,18,527,19,525,21,522,22,520,24,512,28,508,33,505,34,502,37,494,41,492,41,490,43,488,43,484,45,473,45,472,46,451,46'])], 'temp/1763616030_3215042_917855882_da0fa7b7e6b5b551fe26c0ba8713276d.jpg']} ############################### TEST POLYGON ################################ Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.1768021583557129 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu Nov 20 06:20:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3366 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-11-20 06:20:57.601225: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-11-20 06:20:57.626509: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2025-11-20 06:20:57.628679: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f61f4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-11-20 06:20:57.628741: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-11-20 06:20:57.632702: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-11-20 06:20:57.896631: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1242d2f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-11-20 06:20:57.896669: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-11-20 06:20:57.897390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 06:20:57.897731: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 06:20:57.900028: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 06:20:57.902143: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 06:20:57.902617: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 06:20:57.904951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 06:20:57.906039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 06:20:57.912570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 06:20:57.914189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 06:20:57.914330: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 06:20:57.915200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-11-20 06:20:57.915226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-11-20 06:20:57.915241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-11-20 06:20:57.916722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2914 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-11-20 06:20:58.058793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 06:20:58.058940: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 06:20:58.058966: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 06:20:58.058988: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 06:20:58.059009: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 06:20:58.059030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 06:20:58.059050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 06:20:58.059071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 06:20:58.060075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 06:20:58.061226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-11-20 06:20:58.061271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-11-20 06:20:58.061296: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 06:20:58.061319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-11-20 06:20:58.061343: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-11-20 06:20:58.061365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-11-20 06:20:58.061388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-11-20 06:20:58.061411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 06:20:58.062239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-11-20 06:20:58.062284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-11-20 06:20:58.062296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-11-20 06:20:58.062304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-11-20 06:20:58.063161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2914 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-11-20 06:21:07.737471: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-11-20 06:21:07.910818: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-11-20 06:21:09.228309: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.228907: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.11G (2268581120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.228936: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.229521: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.229536: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.236567: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.236586: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.237168: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.237193: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.243688: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.243708: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.244289: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.244305: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.274963: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.274992: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.275578: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.275593: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.281379: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.281398: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.281979: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.281993: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-11-20 06:21:09.315678: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.316280: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.318154: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.318769: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.362564: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.363185: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.365421: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.366014: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.373866: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.374472: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.379302: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.379894: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.392433: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.393019: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.394814: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.395453: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.401831: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.402428: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.404297: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.404871: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.411505: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.412088: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.413793: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.414394: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.445495: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.446072: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.446666: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.447239: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.451270: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.451844: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.468067: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.468642: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.469211: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.469779: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.482117: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.482725: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.483297: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.483865: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.488057: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.488634: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.493114: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.493691: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.505446: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.506019: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.510063: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.510690: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.511328: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.511927: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.532683: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.533260: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.533845: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.534445: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.535021: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.535593: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.549910: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.550491: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.568897: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.568961: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-11-20 06:21:09.570038: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.571127: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.578982: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.579921: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.580568: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.581161: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.589058: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.589685: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.604477: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.605276: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.606069: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.606858: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.611121: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.611904: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.612683: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.613476: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.614717: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.624587: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.625163: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.635315: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.635889: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.636465: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.637034: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.637610: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-11-20 06:21:09.638178: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.35G (2520645632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (2448, 2448, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 2448.00000 nb d'objets trouves : 1 Detection mask done ! Trying to reset tf kernel 3318248 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2173 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3366 list_Values should be empty [] ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.3031342029571533 nb_pixel_total : 3698264 time to create 1 rle with new method : 0.5567619800567627 length of segment : 2043 time spent for convertir_results : 2.6284220218658447 time spend for datou_step_exec : 21.840706825256348 time spend to save output : 3.147125244140625e-05 total time spend for step 1 : 21.84073829650879 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 725 chid ids of type : 445 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.015616893768310547 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'917877156': [[(917877156, 492601069, 445, 0, 2283, 103, 2222, 0.98225856, [(692, 110, 11), (1250, 110, 30), (652, 111, 292), (1202, 111, 135), (614, 112, 379), (1078, 112, 351), (526, 113, 909), (519, 114, 923), (512, 115, 936), (505, 116, 949), (499, 117, 961), (493, 118, 973), (487, 119, 984), (481, 120, 995), (476, 121, 1005), (471, 122, 1015), (466, 123, 1025), (461, 124, 1034), (456, 125, 1044), (451, 126, 1053), (447, 127, 1061), (443, 128, 1073), (438, 129, 1088), (434, 130, 1100), (430, 131, 1113), (427, 132, 1124), (423, 133, 1136), (419, 134, 1148), (416, 135, 1159), (413, 136, 1169), (409, 137, 1180), (406, 138, 1187), (403, 139, 1194), (400, 140, 1200), (397, 141, 1206), (394, 142, 1212), (391, 143, 1219), (388, 144, 1225), (385, 145, 1232), (382, 146, 1238), (378, 147, 1246), (375, 148, 1253), (371, 149, 1261), (368, 150, 1268), (365, 151, 1275), (363, 152, 1281), (360, 153, 1289), (358, 154, 1296), (355, 155, 1303), (353, 156, 1310), (350, 157, 1318), (348, 158, 1325), (345, 159, 1334), (342, 160, 1342), (339, 161, 1350), (336, 162, 1359), (333, 163, 1368), (330, 164, 1377), (327, 165, 1386), (324, 166, 1396), (321, 167, 1405), (318, 168, 1415), (314, 169, 1427), (311, 170, 1437), (307, 171, 1449), (303, 172, 1461), (300, 173, 1473), (296, 174, 1485), (292, 175, 1498), (288, 176, 1511), (284, 177, 1524), (281, 178, 1536), (277, 179, 1549), (274, 180, 1561), (271, 181, 1568), (268, 182, 1576), (266, 183, 1582), (263, 184, 1589), (260, 185, 1596), (258, 186, 1602), (255, 187, 1608), (253, 188, 1614), (250, 189, 1620), (248, 190, 1625), (245, 191, 1632), (243, 192, 1637), (241, 193, 1642), (239, 194, 1647), (237, 195, 1651), (235, 196, 1656), (233, 197, 1661), (231, 198, 1665), (229, 199, 1670), (227, 200, 1674), (225, 201, 1679), (223, 202, 1683), (222, 203, 1686), (220, 204, 1690), (218, 205, 1694), (216, 206, 1699), (215, 207, 1701), (213, 208, 1705), (212, 209, 1707), (210, 210, 1710), (209, 211, 1712), (207, 212, 1716), (206, 213, 1718), (204, 214, 1721), (203, 215, 1723), (202, 216, 1725), (200, 217, 1728), (199, 218, 1730), (198, 219, 1732), (197, 220, 1734), (195, 221, 1737), (194, 222, 1739), (193, 223, 1741), (191, 224, 1744), (190, 225, 1746), (189, 226, 1748), (187, 227, 1751), (186, 228, 1753), (185, 229, 1755), (183, 230, 1758), (182, 231, 1760), (180, 232, 1763), (179, 233, 1765), (178, 234, 1767), (176, 235, 1770), (175, 236, 1773), (173, 237, 1776), (172, 238, 1778), (170, 239, 1781), (169, 240, 1783), (167, 241, 1787), (166, 242, 1789), (164, 243, 1792), (163, 244, 1794), (161, 245, 1798), (159, 246, 1801), (158, 247, 1803), (156, 248, 1807), (155, 249, 1809), (153, 250, 1812), (151, 251, 1816), (150, 252, 1818), (148, 253, 1822), (146, 254, 1825), (145, 255, 1827), (143, 256, 1831), (141, 257, 1834), (140, 258, 1837), (138, 259, 1841), (136, 260, 1844), (134, 261, 1848), (132, 262, 1851), (131, 263, 1854), (129, 264, 1858), (127, 265, 1861), (125, 266, 1865), (123, 267, 1869), (122, 268, 1872), (120, 269, 1875), (119, 270, 1878), (118, 271, 1881), (117, 272, 1883), (115, 273, 1886), (114, 274, 1889), (113, 275, 1891), (112, 276, 1893), (111, 277, 1895), (110, 278, 1897), (109, 279, 1900), (108, 280, 1902), (107, 281, 1904), (106, 282, 1906), (106, 283, 1907), (105, 284, 1909), (104, 285, 1911), (103, 286, 1912), (102, 287, 1914), (101, 288, 1916), (101, 289, 1917), (100, 290, 1919), (99, 291, 1921), (99, 292, 1921), (98, 293, 1923), (98, 294, 1923), (97, 295, 1925), (97, 296, 1925), (97, 297, 1926), (96, 298, 1927), (96, 299, 1928), (96, 300, 1928), (95, 301, 1930), (95, 302, 1930), (94, 303, 1932), (94, 304, 1932), (94, 305, 1933), (93, 306, 1934), (93, 307, 1935), (92, 308, 1936), (92, 309, 1937), (92, 310, 1937), (91, 311, 1939), (91, 312, 1939), (91, 313, 1940), (90, 314, 1941), (90, 315, 1942), (90, 316, 1942), (89, 317, 1944), (89, 318, 1944), (88, 319, 1946), (88, 320, 1946), (88, 321, 1947), (87, 322, 1948), (87, 323, 1949), (87, 324, 1950), (86, 325, 1951), (86, 326, 1952), (86, 327, 1952), (85, 328, 1954), (85, 329, 1954), (85, 330, 1955), (84, 331, 1956), (84, 332, 1957), (84, 333, 1957), (83, 334, 1959), (83, 335, 1959), (82, 336, 1961), (82, 337, 1961), (82, 338, 1962), (81, 339, 1963), (81, 340, 1964), (81, 341, 1964), (80, 342, 1966), (80, 343, 1966), (80, 344, 1967), (79, 345, 1968), (79, 346, 1969), (79, 347, 1969), (78, 348, 1971), (78, 349, 1971), (78, 350, 1972), (78, 351, 1972), (77, 352, 1974), (77, 353, 1974), (77, 354, 1975), (76, 355, 1976), (76, 356, 1977), (76, 357, 1977), (75, 358, 1979), (75, 359, 1980), (75, 360, 1980), (74, 361, 1982), (74, 362, 1982), (74, 363, 1983), (73, 364, 1984), (73, 365, 1985), (73, 366, 1985), (72, 367, 1987), (72, 368, 1987), (72, 369, 1988), (72, 370, 1989), (71, 371, 1990), (71, 372, 1991), (71, 373, 1992), (71, 374, 1993), (70, 375, 1995), (70, 376, 1995), (70, 377, 1996), (70, 378, 1997), (69, 379, 1999), (69, 380, 2000), (69, 381, 2001), (69, 382, 2002), (68, 383, 2004), (68, 384, 2005), (68, 385, 2006), (68, 386, 2007), (67, 387, 2009), (67, 388, 2010), (67, 389, 2011), (66, 390, 2014), (66, 391, 2015), (66, 392, 2016), (66, 393, 2017), (65, 394, 2019), (65, 395, 2020), (65, 396, 2021), (64, 397, 2023), (64, 398, 2024), (64, 399, 2025), (63, 400, 2027), (63, 401, 2028), (63, 402, 2029), (63, 403, 2030), (62, 404, 2032), (62, 405, 2033), (62, 406, 2033), (61, 407, 2035), (61, 408, 2036), (61, 409, 2037), (60, 410, 2038), (60, 411, 2039), (60, 412, 2040), (59, 413, 2042), (59, 414, 2042), (59, 415, 2043), (58, 416, 2045), (58, 417, 2045), (58, 418, 2046), (57, 419, 2048), (57, 420, 2048), (56, 421, 2050), (56, 422, 2050), (56, 423, 2051), (55, 424, 2053), (55, 425, 2053), (55, 426, 2054), (54, 427, 2055), (54, 428, 2056), (53, 429, 2057), (53, 430, 2058), (53, 431, 2058), (52, 432, 2060), (52, 433, 2060), (51, 434, 2062), (51, 435, 2062), (51, 436, 2063), (50, 437, 2064), (50, 438, 2065), (49, 439, 2066), (49, 440, 2067), (48, 441, 2068), (48, 442, 2068), (48, 443, 2069), (47, 444, 2070), (47, 445, 2070), (47, 446, 2070), (47, 447, 2071), (46, 448, 2072), (46, 449, 2072), (46, 450, 2072), (46, 451, 2072), (46, 452, 2073), (45, 453, 2074), (45, 454, 2074), (45, 455, 2074), (45, 456, 2074), (44, 457, 2076), (44, 458, 2076), (44, 459, 2076), (44, 460, 2076), (43, 461, 2077), (43, 462, 2078), (43, 463, 2078), (43, 464, 2078), (42, 465, 2079), (42, 466, 2079), (42, 467, 2080), (42, 468, 2080), (41, 469, 2081), (41, 470, 2081), (41, 471, 2082), (41, 472, 2082), (40, 473, 2083), (40, 474, 2083), (40, 475, 2083), (40, 476, 2084), (39, 477, 2085), (39, 478, 2085), (39, 479, 2085), (39, 480, 2086), (38, 481, 2087), (38, 482, 2087), (38, 483, 2087), (38, 484, 2087), (37, 485, 2089), (37, 486, 2089), (37, 487, 2089), (37, 488, 2089), (36, 489, 2091), (36, 490, 2091), (36, 491, 2091), (36, 492, 2091), (36, 493, 2092), (35, 494, 2093), (35, 495, 2093), (35, 496, 2093), (35, 497, 2094), (34, 498, 2095), (34, 499, 2095), (34, 500, 2095), (34, 501, 2096), (34, 502, 2096), (33, 503, 2097), (33, 504, 2097), (33, 505, 2098), (33, 506, 2098), (33, 507, 2098), (32, 508, 2099), (32, 509, 2100), (32, 510, 2100), (32, 511, 2100), (32, 512, 2100), (31, 513, 2102), (31, 514, 2102), (31, 515, 2102), (31, 516, 2102), (31, 517, 2103), (30, 518, 2104), (30, 519, 2104), (30, 520, 2104), (30, 521, 2105), (30, 522, 2105), (30, 523, 2105), (30, 524, 2105), (29, 525, 2106), (29, 526, 2106), (29, 527, 2107), (29, 528, 2107), (29, 529, 2107), (29, 530, 2107), (29, 531, 2107), (29, 532, 2108), (29, 533, 2108), (28, 534, 2109), (28, 535, 2109), (28, 536, 2109), (28, 537, 2110), (28, 538, 2110), (28, 539, 2110), (28, 540, 2110), (28, 541, 2110), (28, 542, 2110), (28, 543, 2111), (27, 544, 2112), (27, 545, 2112), (27, 546, 2112), (27, 547, 2112), (27, 548, 2113), (27, 549, 2113), (27, 550, 2113), (27, 551, 2113), (27, 552, 2113), (26, 553, 2115), (26, 554, 2115), (26, 555, 2115), (26, 556, 2115), (26, 557, 2115), (26, 558, 2116), (26, 559, 2116), (26, 560, 2116), (26, 561, 2116), (26, 562, 2117), (25, 563, 2118), (25, 564, 2118), (25, 565, 2118), (25, 566, 2118), (25, 567, 2119), (25, 568, 2119), (25, 569, 2119), (25, 570, 2119), (25, 571, 2120), (25, 572, 2120), (24, 573, 2121), (24, 574, 2121), (24, 575, 2121), (24, 576, 2122), (24, 577, 2122), (24, 578, 2122), (24, 579, 2122), (24, 580, 2123), (24, 581, 2123), (24, 582, 2123), (24, 583, 2123), (23, 584, 2125), (23, 585, 2125), (23, 586, 2125), (23, 587, 2125), (23, 588, 2126), (23, 589, 2126), (23, 590, 2126), (23, 591, 2126), (23, 592, 2127), (23, 593, 2127), (23, 594, 2127), (22, 595, 2128), (22, 596, 2128), (22, 597, 2129), (22, 598, 2129), (22, 599, 2129), (22, 600, 2129), (22, 601, 2129), (22, 602, 2129), (22, 603, 2129), (22, 604, 2130), (22, 605, 2130), (22, 606, 2130), (22, 607, 2130), (22, 608, 2130), (22, 609, 2130), (22, 610, 2131), (22, 611, 2131), (21, 612, 2132), (21, 613, 2132), (21, 614, 2132), (21, 615, 2132), (21, 616, 2133), (21, 617, 2133), (21, 618, 2133), (21, 619, 2133), (21, 620, 2133), (21, 621, 2134), (21, 622, 2134), (21, 623, 2134), (21, 624, 2134), (21, 625, 2134), (21, 626, 2134), (21, 627, 2135), (21, 628, 2135), (21, 629, 2135), (20, 630, 2136), (20, 631, 2136), (20, 632, 2136), (20, 633, 2137), (20, 634, 2137), (20, 635, 2137), (20, 636, 2137), (20, 637, 2137), (20, 638, 2138), (20, 639, 2138), (20, 640, 2138), (20, 641, 2138), (20, 642, 2138), (20, 643, 2139), (20, 644, 2139), (20, 645, 2139), (20, 646, 2139), (20, 647, 2139), (19, 648, 2141), (19, 649, 2141), (19, 650, 2141), (19, 651, 2141), (19, 652, 2141), (19, 653, 2142), (19, 654, 2142), (19, 655, 2142), (19, 656, 2142), (19, 657, 2142), (19, 658, 2143), (19, 659, 2143), (19, 660, 2143), (19, 661, 2143), (19, 662, 2143), (19, 663, 2144), (19, 664, 2144), (19, 665, 2144), (19, 666, 2144), (18, 667, 2145), (18, 668, 2145), (18, 669, 2146), (18, 670, 2146), (18, 671, 2146), (19, 672, 2145), (19, 673, 2145), (19, 674, 2144), (19, 675, 2144), (19, 676, 2144), (19, 677, 2144), (19, 678, 2144), (19, 679, 2144), (19, 680, 2144), (19, 681, 2144), (19, 682, 2144), (20, 683, 2142), (20, 684, 2142), (20, 685, 2142), (20, 686, 2142), (20, 687, 2142), (20, 688, 2142), (20, 689, 2142), (20, 690, 2142), (20, 691, 2142), (20, 692, 2141), (21, 693, 2140), (21, 694, 2140), (21, 695, 2140), (21, 696, 2140), (21, 697, 2140), (21, 698, 2140), (21, 699, 2140), (21, 700, 2140), (21, 701, 2139), (22, 702, 2138), (22, 703, 2138), (22, 704, 2138), (22, 705, 2138), (22, 706, 2138), (22, 707, 2138), (22, 708, 2138), (22, 709, 2138), (22, 710, 2137), (22, 711, 2137), (23, 712, 2136), (23, 713, 2136), (23, 714, 2136), (23, 715, 2136), (23, 716, 2136), (23, 717, 2136), (23, 718, 2136), (23, 719, 2136), (23, 720, 2135), (24, 721, 2134), (24, 722, 2134), (24, 723, 2134), (24, 724, 2134), (24, 725, 2134), (24, 726, 2134), (24, 727, 2134), (24, 728, 2134), (24, 729, 2133), (25, 730, 2132), (25, 731, 2132), (25, 732, 2132), (25, 733, 2132), (25, 734, 2132), (25, 735, 2132), (25, 736, 2132), (25, 737, 2132), (26, 738, 2131), (26, 739, 2130), (26, 740, 2130), (26, 741, 2130), (26, 742, 2130), (26, 743, 2130), (26, 744, 2130), (26, 745, 2130), (26, 746, 2130), (26, 747, 2130), (26, 748, 2130), (26, 749, 2129), (26, 750, 2129), (26, 751, 2129), (26, 752, 2129), (26, 753, 2129), (26, 754, 2129), (26, 755, 2129), (26, 756, 2129), (26, 757, 2129), (26, 758, 2129), (26, 759, 2128), (26, 760, 2128), (26, 761, 2128), (26, 762, 2128), (26, 763, 2128), (26, 764, 2128), (26, 765, 2128), (26, 766, 2128), (26, 767, 2128), (26, 768, 2128), (26, 769, 2128), (26, 770, 2127), (26, 771, 2127), (26, 772, 2127), (26, 773, 2127), (26, 774, 2127), (26, 775, 2127), (26, 776, 2127), (26, 777, 2127), (26, 778, 2127), (26, 779, 2127), (26, 780, 2127), (26, 781, 2126), (26, 782, 2126), (26, 783, 2126), (26, 784, 2126), (26, 785, 2126), (26, 786, 2126), (26, 787, 2126), (26, 788, 2126), (26, 789, 2126), (26, 790, 2126), (26, 791, 2126), (26, 792, 2125), (25, 793, 2126), (25, 794, 2126), (25, 795, 2126), (25, 796, 2126), (25, 797, 2126), (25, 798, 2126), (25, 799, 2126), (25, 800, 2126), (25, 801, 2126), (25, 802, 2126), (25, 803, 2126), (25, 804, 2125), (25, 805, 2125), (25, 806, 2125), (25, 807, 2125), (25, 808, 2125), (25, 809, 2125), (25, 810, 2125), (25, 811, 2125), (25, 812, 2125), (25, 813, 2125), (25, 814, 2125), (25, 815, 2125), (25, 816, 2124), (25, 817, 2124), (25, 818, 2124), (25, 819, 2124), (25, 820, 2124), (25, 821, 2124), (25, 822, 2124), (25, 823, 2123), (25, 824, 2123), (25, 825, 2123), (25, 826, 2122), (25, 827, 2122), (25, 828, 2122), (25, 829, 2121), (25, 830, 2121), (26, 831, 2120), (26, 832, 2119), (26, 833, 2119), (26, 834, 2119), (26, 835, 2119), (26, 836, 2118), (26, 837, 2118), (26, 838, 2118), (26, 839, 2117), (26, 840, 2117), (26, 841, 2117), (26, 842, 2116), (26, 843, 2116), (26, 844, 2116), (26, 845, 2116), (27, 846, 2114), (27, 847, 2114), (27, 848, 2114), (27, 849, 2113), (27, 850, 2113), (27, 851, 2113), (27, 852, 2113), (27, 853, 2112), (27, 854, 2112), (27, 855, 2112), (27, 856, 2112), (27, 857, 2111), (27, 858, 2111), (27, 859, 2111), (28, 860, 2110), (28, 861, 2109), (28, 862, 2109), (28, 863, 2109), (28, 864, 2109), (28, 865, 2108), (28, 866, 2108), (28, 867, 2108), (28, 868, 2108), (28, 869, 2107), (28, 870, 2107), (28, 871, 2107), (28, 872, 2107), (28, 873, 2106), (29, 874, 2105), (29, 875, 2105), (29, 876, 2105), (29, 877, 2105), (29, 878, 2104), (29, 879, 2104), (29, 880, 2104), (29, 881, 2104), (29, 882, 2103), (29, 883, 2103), (29, 884, 2103), (29, 885, 2103), (29, 886, 2103), (29, 887, 2102), (30, 888, 2101), (30, 889, 2101), (30, 890, 2101), (30, 891, 2101), (30, 892, 2100), (30, 893, 2100), (30, 894, 2100), (30, 895, 2100), (30, 896, 2100), (30, 897, 2099), (30, 898, 2099), (30, 899, 2099), (30, 900, 2099), (30, 901, 2099), (30, 902, 2099), (30, 903, 2099), (30, 904, 2099), (30, 905, 2099), (30, 906, 2098), (30, 907, 2098), (30, 908, 2098), (30, 909, 2098), (30, 910, 2098), (30, 911, 2098), (29, 912, 2099), (29, 913, 2099), (29, 914, 2099), (29, 915, 2098), (29, 916, 2098), (29, 917, 2098), (29, 918, 2098), (29, 919, 2098), (29, 920, 2098), (29, 921, 2098), (29, 922, 2098), (29, 923, 2098), (29, 924, 2098), (29, 925, 2097), (29, 926, 2097), (29, 927, 2097), (29, 928, 2097), (29, 929, 2097), (29, 930, 2097), (29, 931, 2097), (29, 932, 2097), (28, 933, 2098), (28, 934, 2098), (28, 935, 2097), (28, 936, 2097), (28, 937, 2097), (28, 938, 2097), (28, 939, 2097), (28, 940, 2097), (28, 941, 2097), (28, 942, 2097), (28, 943, 2097), (28, 944, 2097), (28, 945, 2097), (28, 946, 2096), (28, 947, 2096), (28, 948, 2096), (28, 949, 2096), (28, 950, 2096), (28, 951, 2096), (28, 952, 2096), (28, 953, 2096), (28, 954, 2096), (27, 955, 2097), (27, 956, 2096), (27, 957, 2096), (27, 958, 2096), (27, 959, 2096), (27, 960, 2096), (27, 961, 2096), (27, 962, 2096), (27, 963, 2096), (27, 964, 2096), (27, 965, 2096), (27, 966, 2096), (27, 967, 2095), (27, 968, 2095), (27, 969, 2095), (27, 970, 2095), (27, 971, 2095), (27, 972, 2095), (27, 973, 2095), (27, 974, 2095), (27, 975, 2094), (27, 976, 2094), (27, 977, 2094), (27, 978, 2094), (27, 979, 2093), (27, 980, 2093), (27, 981, 2093), (27, 982, 2093), (27, 983, 2092), (27, 984, 2092), (27, 985, 2092), (27, 986, 2091), (27, 987, 2091), (27, 988, 2091), (27, 989, 2091), (27, 990, 2090), (27, 991, 2090), (27, 992, 2090), (28, 993, 2088), (28, 994, 2088), (28, 995, 2088), (28, 996, 2088), (28, 997, 2087), (28, 998, 2087), (28, 999, 2087), (28, 1000, 2086), (28, 1001, 2086), (28, 1002, 2086), (28, 1003, 2085), (28, 1004, 2085), (28, 1005, 2085), (28, 1006, 2084), (28, 1007, 2084), (28, 1008, 2084), (28, 1009, 2083), (28, 1010, 2083), (28, 1011, 2083), (28, 1012, 2082), (28, 1013, 2082), (28, 1014, 2081), (28, 1015, 2081), (28, 1016, 2081), (29, 1017, 2079), (29, 1018, 2079), (29, 1019, 2079), (29, 1020, 2078), (29, 1021, 2078), (29, 1022, 2077), (29, 1023, 2077), (29, 1024, 2077), (29, 1025, 2076), (29, 1026, 2076), (29, 1027, 2075), (29, 1028, 2075), (29, 1029, 2075), (29, 1030, 2074), (29, 1031, 2074), (29, 1032, 2073), (29, 1033, 2073), (29, 1034, 2072), (29, 1035, 2072), (29, 1036, 2071), (29, 1037, 2071), (29, 1038, 2070), (30, 1039, 2069), (30, 1040, 2069), (30, 1041, 2068), (30, 1042, 2068), (30, 1043, 2067), (30, 1044, 2067), (30, 1045, 2066), (30, 1046, 2066), (30, 1047, 2065), (30, 1048, 2064), (30, 1049, 2064), (30, 1050, 2063), (30, 1051, 2062), (30, 1052, 2062), (30, 1053, 2061), (30, 1054, 2060), (30, 1055, 2059), (30, 1056, 2058), (30, 1057, 2058), (30, 1058, 2057), (30, 1059, 2056), (30, 1060, 2055), (30, 1061, 2054), (30, 1062, 2053), (30, 1063, 2052), (30, 1064, 2051), (30, 1065, 2050), (30, 1066, 2049), (30, 1067, 2048), (30, 1068, 2047), (30, 1069, 2046), (30, 1070, 2045), (30, 1071, 2044), (30, 1072, 2043), (30, 1073, 2042), (30, 1074, 2042), (29, 1075, 2042), (29, 1076, 2041), (29, 1077, 2040), (29, 1078, 2039), (29, 1079, 2038), (29, 1080, 2037), (29, 1081, 2037), (29, 1082, 2036), (29, 1083, 2035), (29, 1084, 2034), (29, 1085, 2034), (29, 1086, 2033), (29, 1087, 2032), (29, 1088, 2032), (29, 1089, 2031), (29, 1090, 2030), (29, 1091, 2030), (29, 1092, 2029), (29, 1093, 2028), (29, 1094, 2028), (29, 1095, 2027), (29, 1096, 2027), (29, 1097, 2026), (29, 1098, 2025), (29, 1099, 2025), (29, 1100, 2024), (29, 1101, 2024), (29, 1102, 2023), (29, 1103, 2023), (29, 1104, 2022), (29, 1105, 2022), (29, 1106, 2021), (29, 1107, 2021), (29, 1108, 2020), (29, 1109, 2020), (29, 1110, 2019), (29, 1111, 2019), (29, 1112, 2018), (29, 1113, 2018), (29, 1114, 2017), (29, 1115, 2017), (29, 1116, 2016), (29, 1117, 2016), (29, 1118, 2015), (29, 1119, 2015), (29, 1120, 2015), (29, 1121, 2014), (29, 1122, 2014), (29, 1123, 2013), (29, 1124, 2013), (29, 1125, 2013), (29, 1126, 2012), (29, 1127, 2012), (29, 1128, 2012), (29, 1129, 2011), (29, 1130, 2011), (29, 1131, 2011), (29, 1132, 2010), (29, 1133, 2010), (29, 1134, 2010), (29, 1135, 2009), (29, 1136, 2009), (29, 1137, 2009), (29, 1138, 2008), (29, 1139, 2008), (29, 1140, 2008), (29, 1141, 2007), (29, 1142, 2007), (29, 1143, 2006), (29, 1144, 2006), (29, 1145, 2006), (29, 1146, 2005), (29, 1147, 2005), (29, 1148, 2005), (29, 1149, 2004), (29, 1150, 2004), (29, 1151, 2003), (29, 1152, 2003), (29, 1153, 2003), (29, 1154, 2002), (29, 1155, 2002), (29, 1156, 2001), (29, 1157, 2001), (29, 1158, 2001), (29, 1159, 2000), (29, 1160, 2000), (29, 1161, 1999), (29, 1162, 1999), (29, 1163, 1998), (29, 1164, 1998), (29, 1165, 1997), (29, 1166, 1997), (29, 1167, 1997), (29, 1168, 1996), (29, 1169, 1996), (29, 1170, 1995), (29, 1171, 1995), (29, 1172, 1994), (29, 1173, 1994), (29, 1174, 1993), (29, 1175, 1993), (29, 1176, 1992), (29, 1177, 1992), (29, 1178, 1991), (29, 1179, 1991), (29, 1180, 1990), (29, 1181, 1990), (29, 1182, 1989), (29, 1183, 1989), (29, 1184, 1988), (29, 1185, 1988), (29, 1186, 1987), (29, 1187, 1987), (29, 1188, 1986), (29, 1189, 1986), (29, 1190, 1985), (29, 1191, 1984), (29, 1192, 1984), (29, 1193, 1983), (29, 1194, 1983), (29, 1195, 1982), (29, 1196, 1981), (29, 1197, 1981), (29, 1198, 1980), (29, 1199, 1980), (29, 1200, 1979), (29, 1201, 1978), (29, 1202, 1978), (29, 1203, 1977), (29, 1204, 1976), (29, 1205, 1975), (29, 1206, 1975), (29, 1207, 1974), (29, 1208, 1973), (29, 1209, 1972), (29, 1210, 1971), (29, 1211, 1971), (29, 1212, 1970), (29, 1213, 1969), (29, 1214, 1968), (29, 1215, 1967), (29, 1216, 1966), (29, 1217, 1965), (29, 1218, 1964), (29, 1219, 1964), (29, 1220, 1963), (29, 1221, 1962), (29, 1222, 1961), (29, 1223, 1960), (29, 1224, 1960), (29, 1225, 1959), (29, 1226, 1958), (29, 1227, 1957), (29, 1228, 1957), (29, 1229, 1956), (29, 1230, 1955), (29, 1231, 1955), (29, 1232, 1954), (29, 1233, 1954), (29, 1234, 1953), (29, 1235, 1952), (29, 1236, 1952), (29, 1237, 1951), (29, 1238, 1950), (29, 1239, 1950), (29, 1240, 1949), (29, 1241, 1949), (29, 1242, 1948), (29, 1243, 1948), (29, 1244, 1947), (29, 1245, 1947), (29, 1246, 1946), (29, 1247, 1946), (29, 1248, 1945), (29, 1249, 1945), (29, 1250, 1944), (29, 1251, 1944), (29, 1252, 1943), (29, 1253, 1943), (29, 1254, 1942), (29, 1255, 1942), (30, 1256, 1940), (30, 1257, 1940), (30, 1258, 1940), (30, 1259, 1939), (30, 1260, 1939), (30, 1261, 1938), (30, 1262, 1938), (30, 1263, 1937), (30, 1264, 1937), (30, 1265, 1937), (30, 1266, 1936), (30, 1267, 1936), (30, 1268, 1936), (30, 1269, 1935), (30, 1270, 1935), (30, 1271, 1934), (30, 1272, 1934), (30, 1273, 1934), (30, 1274, 1933), (30, 1275, 1933), (30, 1276, 1933), (30, 1277, 1933), (30, 1278, 1932), (30, 1279, 1932), (30, 1280, 1932), (30, 1281, 1932), (30, 1282, 1931), (30, 1283, 1931), (30, 1284, 1931), (30, 1285, 1931), (30, 1286, 1930), (30, 1287, 1930), (30, 1288, 1930), (30, 1289, 1930), (30, 1290, 1929), (30, 1291, 1929), (30, 1292, 1929), (30, 1293, 1929), (30, 1294, 1928), (30, 1295, 1928), (30, 1296, 1928), (30, 1297, 1928), (30, 1298, 1927), (30, 1299, 1927), (31, 1300, 1926), (31, 1301, 1926), (31, 1302, 1925), (31, 1303, 1925), (31, 1304, 1925), (31, 1305, 1925), (31, 1306, 1924), (31, 1307, 1924), (31, 1308, 1924), (31, 1309, 1924), (31, 1310, 1923), (31, 1311, 1923), (31, 1312, 1923), (31, 1313, 1923), (31, 1314, 1922), (31, 1315, 1922), (31, 1316, 1922), (31, 1317, 1921), (31, 1318, 1921), (31, 1319, 1921), (31, 1320, 1921), (31, 1321, 1920), (31, 1322, 1920), (31, 1323, 1920), (32, 1324, 1918), (32, 1325, 1918), (32, 1326, 1918), (32, 1327, 1918), (32, 1328, 1917), (32, 1329, 1917), (32, 1330, 1917), (32, 1331, 1916), (32, 1332, 1916), (32, 1333, 1916), (32, 1334, 1915), (32, 1335, 1915), (32, 1336, 1915), (32, 1337, 1915), (32, 1338, 1914), (32, 1339, 1914), (32, 1340, 1914), (32, 1341, 1913), (32, 1342, 1913), (32, 1343, 1913), (32, 1344, 1912), (32, 1345, 1912), (32, 1346, 1912), (32, 1347, 1911), (33, 1348, 1910), (33, 1349, 1910), (33, 1350, 1909), (33, 1351, 1909), (33, 1352, 1909), (33, 1353, 1908), (33, 1354, 1908), (33, 1355, 1907), (33, 1356, 1906), (33, 1357, 1906), (33, 1358, 1905), (33, 1359, 1905), (33, 1360, 1904), (33, 1361, 1904), (34, 1362, 1902), (34, 1363, 1901), (34, 1364, 1901), (34, 1365, 1900), (34, 1366, 1900), (34, 1367, 1899), (34, 1368, 1898), (34, 1369, 1898), (34, 1370, 1897), (34, 1371, 1896), (34, 1372, 1896), (34, 1373, 1895), (35, 1374, 1893), (35, 1375, 1892), (35, 1376, 1892), (35, 1377, 1891), (35, 1378, 1890), (35, 1379, 1889), (35, 1380, 1888), (35, 1381, 1888), (35, 1382, 1887), (35, 1383, 1886), (35, 1384, 1885), (36, 1385, 1883), (36, 1386, 1882), (36, 1387, 1881), (36, 1388, 1880), (36, 1389, 1879), (36, 1390, 1878), (36, 1391, 1877), (36, 1392, 1876), (36, 1393, 1875), (36, 1394, 1875), (36, 1395, 1874), (37, 1396, 1872), (37, 1397, 1871), (37, 1398, 1870), (37, 1399, 1870), (37, 1400, 1869), (37, 1401, 1868), (37, 1402, 1867), (37, 1403, 1867), (37, 1404, 1866), (37, 1405, 1865), (37, 1406, 1864), (38, 1407, 1863), (38, 1408, 1862), (38, 1409, 1862), (38, 1410, 1861), (38, 1411, 1860), (38, 1412, 1860), (38, 1413, 1859), (38, 1414, 1858), (38, 1415, 1858), (38, 1416, 1857), (38, 1417, 1857), (39, 1418, 1855), (39, 1419, 1855), (39, 1420, 1854), (39, 1421, 1854), (39, 1422, 1853), (39, 1423, 1853), (39, 1424, 1852), (39, 1425, 1852), (39, 1426, 1851), (39, 1427, 1851), (39, 1428, 1850), (39, 1429, 1850), (39, 1430, 1850), (39, 1431, 1850), (39, 1432, 1849), (40, 1433, 1848), (40, 1434, 1848), (40, 1435, 1848), (40, 1436, 1848), (40, 1437, 1847), (40, 1438, 1847), (40, 1439, 1847), (40, 1440, 1847), (40, 1441, 1847), (40, 1442, 1846), (40, 1443, 1846), (40, 1444, 1846), (40, 1445, 1846), (40, 1446, 1845), (40, 1447, 1845), (40, 1448, 1845), (40, 1449, 1845), (40, 1450, 1845), (40, 1451, 1844), (40, 1452, 1844), (40, 1453, 1844), (40, 1454, 1844), (40, 1455, 1844), (40, 1456, 1843), (40, 1457, 1843), (40, 1458, 1843), (40, 1459, 1843), (40, 1460, 1843), (40, 1461, 1843), (40, 1462, 1842), (40, 1463, 1842), (40, 1464, 1842), (40, 1465, 1842), (40, 1466, 1842), (40, 1467, 1841), (40, 1468, 1841), (40, 1469, 1841), (40, 1470, 1841), (40, 1471, 1841), (40, 1472, 1840), (40, 1473, 1840), (40, 1474, 1840), (40, 1475, 1840), (40, 1476, 1840), (40, 1477, 1840), (41, 1478, 1838), (41, 1479, 1838), (41, 1480, 1838), (41, 1481, 1838), (41, 1482, 1838), (41, 1483, 1837), (41, 1484, 1837), (41, 1485, 1837), (41, 1486, 1837), (41, 1487, 1837), (41, 1488, 1837), (41, 1489, 1836), (41, 1490, 1836), (41, 1491, 1836), (41, 1492, 1836), (41, 1493, 1836), (41, 1494, 1836), (41, 1495, 1835), (41, 1496, 1835), (41, 1497, 1835), (41, 1498, 1835), (41, 1499, 1835), (41, 1500, 1835), (41, 1501, 1834), (41, 1502, 1834), (41, 1503, 1834), (41, 1504, 1834), (41, 1505, 1834), (41, 1506, 1834), (41, 1507, 1834), (41, 1508, 1833), (41, 1509, 1833), (41, 1510, 1833), (41, 1511, 1833), (42, 1512, 1832), (42, 1513, 1832), (42, 1514, 1832), (42, 1515, 1832), (42, 1516, 1831), (42, 1517, 1831), (42, 1518, 1831), (42, 1519, 1831), (42, 1520, 1831), (42, 1521, 1831), (42, 1522, 1831), (42, 1523, 1830), (42, 1524, 1830), (42, 1525, 1830), (42, 1526, 1830), (42, 1527, 1830), (42, 1528, 1830), (42, 1529, 1830), (42, 1530, 1830), (42, 1531, 1829), (42, 1532, 1829), (42, 1533, 1829), (42, 1534, 1829), (43, 1535, 1828), (43, 1536, 1828), (43, 1537, 1828), (43, 1538, 1827), (43, 1539, 1827), (43, 1540, 1827), (43, 1541, 1827), (43, 1542, 1827), (43, 1543, 1827), (43, 1544, 1827), (43, 1545, 1826), (43, 1546, 1826), (43, 1547, 1826), (43, 1548, 1826), (43, 1549, 1826), (43, 1550, 1826), (43, 1551, 1826), (43, 1552, 1825), (43, 1553, 1825), (43, 1554, 1825), (43, 1555, 1825), (43, 1556, 1825), (43, 1557, 1825), (43, 1558, 1825), (44, 1559, 1823), (44, 1560, 1823), (44, 1561, 1823), (44, 1562, 1823), (44, 1563, 1823), (44, 1564, 1823), (44, 1565, 1822), (44, 1566, 1822), (44, 1567, 1822), (44, 1568, 1822), (44, 1569, 1822), (44, 1570, 1822), (44, 1571, 1822), (44, 1572, 1821), (44, 1573, 1821), (44, 1574, 1821), (44, 1575, 1821), (44, 1576, 1821), (44, 1577, 1821), (44, 1578, 1821), (44, 1579, 1820), (44, 1580, 1820), (44, 1581, 1820), (44, 1582, 1820), (44, 1583, 1820), (44, 1584, 1820), (44, 1585, 1819), (44, 1586, 1819), (44, 1587, 1819), (44, 1588, 1819), (44, 1589, 1819), (44, 1590, 1819), (44, 1591, 1819), (44, 1592, 1818), (43, 1593, 1819), (43, 1594, 1819), (43, 1595, 1819), (43, 1596, 1819), (43, 1597, 1819), (43, 1598, 1818), (43, 1599, 1818), (43, 1600, 1818), (43, 1601, 1818), (43, 1602, 1818), (43, 1603, 1818), (43, 1604, 1818), (43, 1605, 1817), (43, 1606, 1817), (43, 1607, 1817), (43, 1608, 1817), (43, 1609, 1817), (42, 1610, 1818), (42, 1611, 1817), (42, 1612, 1817), (42, 1613, 1817), (42, 1614, 1817), (42, 1615, 1817), (42, 1616, 1816), (42, 1617, 1816), (42, 1618, 1816), (42, 1619, 1816), (42, 1620, 1816), (42, 1621, 1816), (42, 1622, 1815), (42, 1623, 1815), (42, 1624, 1815), (42, 1625, 1815), (41, 1626, 1816), (41, 1627, 1815), (41, 1628, 1815), (41, 1629, 1815), (41, 1630, 1815), (41, 1631, 1815), (41, 1632, 1815), (41, 1633, 1814), (41, 1634, 1814), (41, 1635, 1814), (41, 1636, 1814), (41, 1637, 1814), (41, 1638, 1813), (41, 1639, 1813), (41, 1640, 1813), (41, 1641, 1813), (40, 1642, 1814), (40, 1643, 1813), (40, 1644, 1813), (40, 1645, 1813), (40, 1646, 1813), (40, 1647, 1813), (40, 1648, 1812), (40, 1649, 1812), (40, 1650, 1812), (40, 1651, 1812), (40, 1652, 1811), (40, 1653, 1811), (40, 1654, 1811), (40, 1655, 1811), (40, 1656, 1810), (40, 1657, 1810), (40, 1658, 1810), (41, 1659, 1809), (41, 1660, 1808), (41, 1661, 1808), (41, 1662, 1808), (41, 1663, 1807), (41, 1664, 1807), (42, 1665, 1806), (42, 1666, 1805), (42, 1667, 1805), (42, 1668, 1805), (42, 1669, 1805), (42, 1670, 1804), (43, 1671, 1803), (43, 1672, 1803), (43, 1673, 1802), (43, 1674, 1802), (43, 1675, 1802), (43, 1676, 1801), (44, 1677, 1800), (44, 1678, 1800), (44, 1679, 1799), (44, 1680, 1799), (44, 1681, 1799), (44, 1682, 1798), (44, 1683, 1798), (45, 1684, 1797), (45, 1685, 1796), (45, 1686, 1796), (45, 1687, 1795), (45, 1688, 1795), (45, 1689, 1795), (46, 1690, 1793), (46, 1691, 1793), (46, 1692, 1793), (46, 1693, 1792), (46, 1694, 1792), (46, 1695, 1791), (47, 1696, 1790), (47, 1697, 1790), (47, 1698, 1789), (47, 1699, 1789), (47, 1700, 1788), (47, 1701, 1788), (47, 1702, 1788), (48, 1703, 1786), (48, 1704, 1786), (48, 1705, 1785), (48, 1706, 1785), (48, 1707, 1784), (48, 1708, 1784), (48, 1709, 1783), (49, 1710, 1782), (49, 1711, 1782), (49, 1712, 1781), (49, 1713, 1781), (49, 1714, 1780), (49, 1715, 1780), (49, 1716, 1780), (50, 1717, 1778), (50, 1718, 1778), (50, 1719, 1777), (50, 1720, 1777), (50, 1721, 1777), (50, 1722, 1776), (50, 1723, 1776), (51, 1724, 1774), (51, 1725, 1774), (51, 1726, 1774), (51, 1727, 1773), (51, 1728, 1773), (51, 1729, 1773), (51, 1730, 1772), (51, 1731, 1772), (51, 1732, 1772), (51, 1733, 1771), (51, 1734, 1771), (51, 1735, 1771), (51, 1736, 1770), (51, 1737, 1770), (51, 1738, 1770), (51, 1739, 1769), (51, 1740, 1769), (51, 1741, 1769), (51, 1742, 1769), (51, 1743, 1768), (51, 1744, 1768), (51, 1745, 1768), (51, 1746, 1767), (51, 1747, 1767), (51, 1748, 1767), (51, 1749, 1766), (51, 1750, 1766), (51, 1751, 1766), (51, 1752, 1766), (51, 1753, 1765), (51, 1754, 1765), (51, 1755, 1765), (51, 1756, 1764), (51, 1757, 1764), (51, 1758, 1764), (51, 1759, 1764), (51, 1760, 1763), (51, 1761, 1763), (51, 1762, 1763), (51, 1763, 1762), (51, 1764, 1762), (51, 1765, 1762), (51, 1766, 1762), (51, 1767, 1761), (51, 1768, 1761), (51, 1769, 1761), (51, 1770, 1761), (51, 1771, 1760), (51, 1772, 1760), (51, 1773, 1760), (51, 1774, 1760), (51, 1775, 1759), (51, 1776, 1759), (51, 1777, 1759), (51, 1778, 1758), (51, 1779, 1758), (51, 1780, 1758), (51, 1781, 1758), (51, 1782, 1757), (51, 1783, 1757), (51, 1784, 1757), (51, 1785, 1757), (51, 1786, 1757), (51, 1787, 1756), (51, 1788, 1756), (51, 1789, 1756), (51, 1790, 1756), (51, 1791, 1755), (51, 1792, 1755), (51, 1793, 1755), (51, 1794, 1755), (51, 1795, 1754), (51, 1796, 1754), (51, 1797, 1754), (51, 1798, 1754), (51, 1799, 1753), (51, 1800, 1753), (51, 1801, 1753), (51, 1802, 1753), (51, 1803, 1753), (51, 1804, 1752), (51, 1805, 1752), (52, 1806, 1751), (52, 1807, 1751), (52, 1808, 1750), (53, 1809, 1749), (53, 1810, 1749), (54, 1811, 1747), (54, 1812, 1747), (54, 1813, 1747), (55, 1814, 1745), (55, 1815, 1745), (55, 1816, 1745), (56, 1817, 1744), (56, 1818, 1743), (57, 1819, 1742), (57, 1820, 1742), (57, 1821, 1741), (58, 1822, 1740), (58, 1823, 1740), (59, 1824, 1738), (59, 1825, 1738), (59, 1826, 1738), (60, 1827, 1737), (60, 1828, 1736), (60, 1829, 1736), (61, 1830, 1735), (61, 1831, 1734), (62, 1832, 1733), (62, 1833, 1733), (62, 1834, 1732), (63, 1835, 1731), (63, 1836, 1731), (63, 1837, 1730), (64, 1838, 1729), (64, 1839, 1729), (65, 1840, 1727), (65, 1841, 1727), (65, 1842, 1727), (66, 1843, 1725), (66, 1844, 1725), (66, 1845, 1725), (67, 1846, 1723), (67, 1847, 1723), (67, 1848, 1723), (68, 1849, 1721), (68, 1850, 1721), (69, 1851, 1720), (69, 1852, 1719), (69, 1853, 1719), (70, 1854, 1718), (70, 1855, 1717), (70, 1856, 1717), (71, 1857, 1716), (71, 1858, 1715), (71, 1859, 1715), (72, 1860, 1714), (72, 1861, 1713), (72, 1862, 1713), (73, 1863, 1711), (73, 1864, 1711), (73, 1865, 1711), (74, 1866, 1709), (74, 1867, 1709), (74, 1868, 1709), (75, 1869, 1707), (75, 1870, 1707), (76, 1871, 1705), (76, 1872, 1705), (76, 1873, 1705), (77, 1874, 1703), (77, 1875, 1703), (77, 1876, 1703), (78, 1877, 1701), (78, 1878, 1701), (78, 1879, 1700), (79, 1880, 1699), (79, 1881, 1699), (79, 1882, 1698), (79, 1883, 1698), (80, 1884, 1696), (80, 1885, 1696), (80, 1886, 1695), (81, 1887, 1694), (81, 1888, 1694), (81, 1889, 1693), (82, 1890, 1692), (82, 1891, 1691), (82, 1892, 1691), (82, 1893, 1690), (83, 1894, 1689), (83, 1895, 1688), (83, 1896, 1688), (84, 1897, 1686), (84, 1898, 1686), (84, 1899, 1685), (85, 1900, 1684), (85, 1901, 1683), (85, 1902, 1683), (86, 1903, 1681), (86, 1904, 1681), (87, 1905, 1679), (87, 1906, 1679), (87, 1907, 1678), (88, 1908, 1676), (88, 1909, 1676), (88, 1910, 1675), (89, 1911, 1674), (89, 1912, 1673), (89, 1913, 1673), (90, 1914, 1671), (90, 1915, 1670), (91, 1916, 1669), (91, 1917, 1668), (91, 1918, 1668), (92, 1919, 1666), (92, 1920, 1665), (93, 1921, 1664), (93, 1922, 1663), (94, 1923, 1661), (94, 1924, 1661), (94, 1925, 1660), (95, 1926, 1658), (95, 1927, 1658), (96, 1928, 1656), (96, 1929, 1655), (97, 1930, 1653), (97, 1931, 1653), (98, 1932, 1651), (98, 1933, 1650), (99, 1934, 1649), (99, 1935, 1648), (100, 1936, 1646), (100, 1937, 1646), (101, 1938, 1644), (101, 1939, 1643), (102, 1940, 1642), (102, 1941, 1641), (103, 1942, 1639), (103, 1943, 1639), (104, 1944, 1637), (104, 1945, 1636), (105, 1946, 1635), (106, 1947, 1633), (106, 1948, 1633), (107, 1949, 1631), (107, 1950, 1630), (108, 1951, 1629), (109, 1952, 1627), (109, 1953, 1627), (110, 1954, 1625), (110, 1955, 1624), (111, 1956, 1623), (112, 1957, 1621), (113, 1958, 1619), (113, 1959, 1619), (114, 1960, 1617), (115, 1961, 1615), (116, 1962, 1613), (116, 1963, 1612), (117, 1964, 1610), (118, 1965, 76), (207, 1965, 1519), (119, 1966, 57), (213, 1966, 1512), (120, 1967, 40), (220, 1967, 1504), (121, 1968, 25), (226, 1968, 1498), (121, 1969, 12), (233, 1969, 1490), (240, 1970, 1482), (248, 1971, 1473), (255, 1972, 1465), (263, 1973, 1456), (271, 1974, 1447), (279, 1975, 1438), (288, 1976, 1428), (297, 1977, 1418), (307, 1978, 1407), (318, 1979, 1395), (330, 1980, 1381), (343, 1981, 1367), (357, 1982, 1352), (369, 1983, 1339), (374, 1984, 1333), (379, 1985, 1327), (385, 1986, 1320), (390, 1987, 1314), (396, 1988, 1306), (401, 1989, 1300), (407, 1990, 1293), (413, 1991, 1286), (418, 1992, 1280), (424, 1993, 1272), (430, 1994, 1265), (436, 1995, 1258), (442, 1996, 1250), (448, 1997, 1243), (453, 1998, 1237), (458, 1999, 1230), (463, 2000, 1224), (468, 2001, 1217), (473, 2002, 1211), (478, 2003, 1205), (482, 2004, 1199), (487, 2005, 1193), (492, 2006, 1186), (497, 2007, 1180), (501, 2008, 1174), (506, 2009, 1167), (510, 2010, 1162), (515, 2011, 1141), (519, 2012, 1116), (524, 2013, 1090), (528, 2014, 1064), (532, 2015, 1055), (536, 2016, 1048), (540, 2017, 1042), (543, 2018, 1036), (547, 2019, 1029), (551, 2020, 1023), (554, 2021, 1017), (557, 2022, 1012), (561, 2023, 1005), (564, 2024, 1000), (567, 2025, 994), (570, 2026, 989), (573, 2027, 984), (576, 2028, 979), (579, 2029, 973), (582, 2030, 968), (585, 2031, 963), (588, 2032, 958), (590, 2033, 954), (592, 2034, 950), (594, 2035, 947), (597, 2036, 942), (599, 2037, 938), (601, 2038, 934), (604, 2039, 929), (606, 2040, 925), (609, 2041, 920), (611, 2042, 915), (613, 2043, 911), (616, 2044, 906), (618, 2045, 902), (620, 2046, 897), (622, 2047, 893), (624, 2048, 888), (627, 2049, 882), (629, 2050, 871), (631, 2051, 855), (634, 2052, 838), (636, 2053, 822), (639, 2054, 803), (642, 2055, 785), (644, 2056, 777), (647, 2057, 769), (650, 2058, 760), (653, 2059, 751), (656, 2060, 743), (659, 2061, 734), (662, 2062, 725), (665, 2063, 716), (668, 2064, 707), (671, 2065, 698), (674, 2066, 689), (678, 2067, 678), (681, 2068, 669), (685, 2069, 658), (688, 2070, 648), (692, 2071, 636), (696, 2072, 625), (701, 2073, 613), (705, 2074, 601), (710, 2075, 589), (714, 2076, 577), (718, 2077, 566), (722, 2078, 555), (726, 2079, 544), (730, 2080, 533), (734, 2081, 525), (737, 2082, 518), (741, 2083, 509), (745, 2084, 502), (748, 2085, 495), (752, 2086, 487), (755, 2087, 480), (758, 2088, 474), (762, 2089, 467), (765, 2090, 460), (768, 2091, 454), (771, 2092, 448), (774, 2093, 442), (777, 2094, 436), (780, 2095, 430), (782, 2096, 425), (785, 2097, 419), (788, 2098, 413), (790, 2099, 409), (793, 2100, 403), (795, 2101, 398), (798, 2102, 393), (800, 2103, 388), (802, 2104, 384), (805, 2105, 379), (807, 2106, 375), (809, 2107, 371), (812, 2108, 366), (814, 2109, 361), (817, 2110, 356), (819, 2111, 352), (822, 2112, 346), (825, 2113, 341), (827, 2114, 337), (830, 2115, 331), (833, 2116, 326), (836, 2117, 320), (839, 2118, 315), (842, 2119, 309), (845, 2120, 303), (849, 2121, 297), (852, 2122, 291), (855, 2123, 285), (859, 2124, 278), (864, 2125, 270), (868, 2126, 264), (872, 2127, 257), (876, 2128, 250), (881, 2129, 241), (885, 2130, 234), (890, 2131, 226), (894, 2132, 219), (899, 2133, 211), (904, 2134, 202), (908, 2135, 195), (913, 2136, 184), (918, 2137, 170), (923, 2138, 157), (928, 2139, 143), (933, 2140, 129), (938, 2141, 115), (959, 2142, 85), (980, 2143, 55), (1001, 2144, 25)], ['1025,2144,938,2141,855,2123,688,2070,531,2014,356,1981,206,1964,121,1969,94,1925,51,1805,40,1658,44,1592,40,1433,29,1255,30,911,19,648,29,525,47,444,97,295,120,269,245,191,419,134,526,113,652,111,1428,112,1596,139,1748,171,1834,180,1914,206,2019,291,2064,376,2116,443,2163,673,2148,822,2122,966,2095,1046,2056,1093,2025,1167,1978,1238,1940,1354,1888,1428,1855,1632,1774,1888,1731,1959,1671,2010,1587,2014,1508,2049,1427,2054,1235,2086,1109,2133'])], 'temp/1763616053_3215042_917877156_a9c2d4b99270c9302def4ed40606e685.jpg']} nb pixel non reg : 3692295 nb pixel common : 3678732 proportion of common points : 0.9963266748729449 #&_# TEST FAILED #&_# : tests/mask_test #&_# #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : sam list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.3166968822479248 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:sam Thu Nov 20 06:21:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step sam ! Inside sam : nb paths : 1 (640, 960, 3) ERROR in datou_step_exec, will save and exit ! CUDA out of memory. Tried to allocate 768.00 MiB (GPU 0; 10.76 GiB total capacity; 1.95 GiB already allocated; 292.88 MiB free; 2.04 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2339, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2440, in datou_step_exec return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 396, in datou_step_sam masks = mask_generator.generate(image) File "/home/admin/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/automatic_mask_generator.py", line 163, in generate mask_data = self._generate_masks(image) File "/home/admin/workarea/install/segment-anything/segment_anything/automatic_mask_generator.py", line 206, in _generate_masks crop_data = self._process_crop(image, crop_box, layer_idx, orig_size) File "/home/admin/workarea/install/segment-anything/segment_anything/automatic_mask_generator.py", line 236, in _process_crop self.predictor.set_image(cropped_im) File "/home/admin/workarea/install/segment-anything/segment_anything/predictor.py", line 60, in set_image self.set_torch_image(input_image_torch, image.shape[:2]) File "/home/admin/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/predictor.py", line 89, in set_torch_image self.features = self.model.image_encoder(input_image) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/modeling/image_encoder.py", line 112, in forward x = blk(x) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/modeling/image_encoder.py", line 174, in forward x = self.attn(x) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/modeling/image_encoder.py", line 234, in forward attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W)) File "/home/admin/workarea/install/segment-anything/segment_anything/modeling/image_encoder.py", line 358, in add_decomposed_rel_pos attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :] [1189321094] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.018034934997558594 save_final ERROR in last step sam, CUDA out of memory. Tried to allocate 768.00 MiB (GPU 0; 10.76 GiB total capacity; 1.95 GiB already allocated; 292.88 MiB free; 2.04 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF time spend for datou_step_exec : 9.331957817077637 time spend to save output : 0.030763626098632812 total time spend for step 0 : 9.36272144317627 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : None ERROR nb objects espect : 98 nb_objects detect : 0 ERROR sam FAILED ############################### TEST frcnn ################################ Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : frcnn list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.15404462814331055 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:frcnn Thu Nov 20 06:21:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step Faster rcnn ! To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] WARNING: Logging before InitGoogleLogging() is written to STDERR F1120 06:21:35.455328 3215042 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Aborted (core dumped) No data to report. No data to report. ret : 34304 command : coverage3 html -i --omit=/usr/local/lib/python3.8/dist-packages/*,/home/admin/.local/lib/python3.8/site-packages/*,/usr/lib/python3/dist-packages/* -d htmlcov ret : 256 command : coverage3 report -i -m ret : 256 42.09user 27.23system 1:01:12elapsed 1%CPU (0avgtext+0avgdata 3550788maxresident)k 4626968inputs+4768outputs (24223major+3136242minor)pagefaults 0swaps