python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3139 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.1412959098815918 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Tue Feb 4 18:35:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3139 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-02-04 18:35:30.410756: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-04 18:35:30.439079: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-04 18:35:30.440636: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7cc8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-04 18:35:30.440706: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-04 18:35:30.443877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-04 18:35:30.713372: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d5dfb00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-04 18:35:30.713427: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-04 18:35:30.714398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:35:30.714759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:35:30.717216: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:35:30.719833: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:35:30.720279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:35:30.723051: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:35:30.724403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:35:30.730040: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:35:30.731555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:35:30.731700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:35:30.732346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 18:35:30.732363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 18:35:30.732373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 18:35:30.733341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2687 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-02-04 18:35:31.296299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:35:31.296397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:35:31.296429: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:35:31.296458: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:35:31.296486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:35:31.296514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:35:31.296558: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:35:31.296587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:35:31.297944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:35:31.299304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:35:31.299371: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:35:31.299397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:35:31.299421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:35:31.299444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:35:31.299468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:35:31.299491: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:35:31.299515: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:35:31.300552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:35:31.300581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 18:35:31.300590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 18:35:31.300597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 18:35:31.301496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2687 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-04 18:35:37.920376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:35:38.079864: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:35:39.411522: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.411633: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.412247: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.412264: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.419763: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.419851: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.420456: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.420473: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.427324: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.427379: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.427983: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.427999: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.459440: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.459502: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.460100: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.460117: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.466051: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.466091: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.466703: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.466719: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:39.501721: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.502336: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.504270: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.504866: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.550806: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.551439: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.558243: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.558846: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.563153: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.563758: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.574456: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.575090: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.576663: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.577228: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.582652: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.583235: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.584928: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.585523: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.591225: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.591824: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.593334: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.593882: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.619670: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.620225: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.620770: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.621316: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.624766: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.625315: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.640406: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.640958: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.641504: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.642047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.654069: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.654616: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.655165: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.655707: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.659943: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.660499: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.664926: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.665500: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.677126: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.677677: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.681724: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.682336: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.682952: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.683549: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.704311: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.704910: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.705519: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.706110: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.706660: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.707246: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.721769: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.722320: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.744988: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.745033: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-04 18:35:39.745646: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.746248: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.753939: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.754543: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.762573: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.763190: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.777947: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.778594: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.779219: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.779789: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.783921: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.784510: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.785058: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.785604: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.786744: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.796991: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.797570: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.807859: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.808438: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.808993: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.809558: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.810162: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:39.810755: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (480, 640, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 599274 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1946 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3139 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0005664825439453125 nb_pixel_total : 15552 time to create 1 rle with old method : 0.016901016235351562 length of segment : 256 time for calcul the mask position with numpy : 0.0027036666870117188 nb_pixel_total : 145329 time to create 1 rle with old method : 0.1579148769378662 length of segment : 371 time for calcul the mask position with numpy : 0.00022721290588378906 nb_pixel_total : 14255 time to create 1 rle with old method : 0.01576089859008789 length of segment : 151 time for calcul the mask position with numpy : 0.00011396408081054688 nb_pixel_total : 5613 time to create 1 rle with old method : 0.006661891937255859 length of segment : 48 time for calcul the mask position with numpy : 5.6743621826171875e-05 nb_pixel_total : 1824 time to create 1 rle with old method : 0.0024023056030273438 length of segment : 39 time spent for convertir_results : 0.9770581722259521 time spend for datou_step_exec : 16.26106858253479 time spend to save output : 4.5299530029296875e-05 total time spend for step 1 : 16.26111388206482 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 3260 chid ids of type : 445 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.01265406608581543 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'957285035': [[(957285035, 492601069, 445, 0, 186, 22, 282, 0.9954951, [(140, 26, 6), (135, 27, 15), (133, 28, 18), (131, 29, 22), (126, 30, 28), (10, 31, 1), (120, 31, 35), (8, 32, 13), (27, 32, 3), (115, 32, 41), (7, 33, 52), (109, 33, 48), (6, 34, 70), (103, 34, 55), (5, 35, 154), (4, 36, 155), (3, 37, 156), (3, 38, 156), (3, 39, 156), (2, 40, 157), (2, 41, 157), (2, 42, 157), (2, 43, 157), (2, 44, 157), (2, 45, 157), (1, 46, 158), (1, 47, 158), (1, 48, 158), (1, 49, 157), (1, 50, 157), (1, 51, 156), (1, 52, 156), (1, 53, 155), (1, 54, 154), (1, 55, 152), (1, 56, 149), (1, 57, 145), (1, 58, 141), (1, 59, 136), (1, 60, 133), (1, 61, 130), (1, 62, 127), (1, 63, 126), (1, 64, 124), (1, 65, 123), (1, 66, 121), (1, 67, 120), (1, 68, 118), (1, 69, 117), (1, 70, 116), (1, 71, 115), (1, 72, 114), (1, 73, 113), (1, 74, 112), (1, 75, 111), (1, 76, 110), (1, 77, 108), (1, 78, 108), (1, 79, 107), (1, 80, 106), (1, 81, 105), (2, 82, 104), (2, 83, 103), (2, 84, 103), (2, 85, 102), (2, 86, 102), (2, 87, 101), (2, 88, 100), (2, 89, 99), (2, 90, 99), (2, 91, 98), (2, 92, 97), (2, 93, 96), (2, 94, 95), (2, 95, 93), (2, 96, 91), (2, 97, 90), (2, 98, 89), (2, 99, 87), (2, 100, 86), (2, 101, 86), (2, 102, 85), (2, 103, 84), (2, 104, 83), (2, 105, 83), (2, 106, 82), (2, 107, 81), (2, 108, 80), (2, 109, 80), (2, 110, 79), (2, 111, 78), (2, 112, 77), (2, 113, 76), (1, 114, 76), (1, 115, 75), (1, 116, 74), (1, 117, 73), (1, 118, 72), (1, 119, 71), (1, 120, 71), (1, 121, 70), (1, 122, 69), (1, 123, 69), (1, 124, 68), (1, 125, 68), (1, 126, 67), (1, 127, 67), (1, 128, 66), (1, 129, 66), (1, 130, 66), (1, 131, 65), (1, 132, 65), (1, 133, 64), (1, 134, 63), (1, 135, 63), (1, 136, 62), (1, 137, 61), (1, 138, 60), (1, 139, 60), (1, 140, 59), (1, 141, 58), (1, 142, 58), (1, 143, 57), (1, 144, 56), (1, 145, 56), (1, 146, 55), (1, 147, 54), (1, 148, 54), (1, 149, 53), (1, 150, 52), (1, 151, 52), (1, 152, 51), (1, 153, 50), (1, 154, 49), (1, 155, 48), (1, 156, 47), (1, 157, 46), (1, 158, 45), (1, 159, 45), (1, 160, 44), (1, 161, 43), (1, 162, 42), (1, 163, 41), (1, 164, 41), (1, 165, 40), (1, 166, 40), (1, 167, 39), (1, 168, 38), (1, 169, 37), (1, 170, 36), (1, 171, 35), (1, 172, 34), (1, 173, 34), (1, 174, 33), (1, 175, 33), (1, 176, 32), (1, 177, 32), (1, 178, 32), (1, 179, 32), (1, 180, 31), (1, 181, 31), (1, 182, 31), (1, 183, 30), (1, 184, 30), (1, 185, 30), (1, 186, 29), (1, 187, 29), (1, 188, 29), (1, 189, 28), (1, 190, 28), (1, 191, 27), (1, 192, 27), (1, 193, 26), (1, 194, 26), (1, 195, 26), (1, 196, 26), (1, 197, 26), (1, 198, 26), (1, 199, 26), (1, 200, 25), (1, 201, 25), (1, 202, 25), (1, 203, 25), (1, 204, 25), (1, 205, 25), (1, 206, 25), (1, 207, 25), (1, 208, 25), (1, 209, 25), (1, 210, 25), (1, 211, 25), (1, 212, 25), (1, 213, 25), (1, 214, 25), (1, 215, 25), (1, 216, 25), (1, 217, 25), (1, 218, 25), (1, 219, 25), (1, 220, 24), (1, 221, 24), (1, 222, 24), (1, 223, 24), (1, 224, 24), (1, 225, 24), (1, 226, 25), (1, 227, 25), (1, 228, 25), (2, 229, 24), (2, 230, 24), (2, 231, 24), (2, 232, 23), (2, 233, 23), (2, 234, 23), (2, 235, 23), (2, 236, 23), (2, 237, 23), (2, 238, 23), (2, 239, 23), (2, 240, 23), (2, 241, 23), (2, 242, 23), (2, 243, 23), (2, 244, 23), (2, 245, 23), (2, 246, 23), (2, 247, 23), (2, 248, 23), (2, 249, 24), (2, 250, 24), (2, 251, 23), (2, 252, 23), (2, 253, 23), (2, 254, 23), (2, 255, 23), (2, 256, 23), (2, 257, 23), (2, 258, 23), (2, 259, 23), (2, 260, 23), (2, 261, 23), (3, 262, 22), (3, 263, 22), (3, 264, 22), (3, 265, 22), (4, 266, 21), (4, 267, 21), (5, 268, 20), (5, 269, 20), (6, 270, 19), (7, 271, 17), (8, 272, 16), (8, 273, 16), (9, 274, 13), (11, 275, 9), (15, 276, 2)], ['16,276,8,273,2,261,2,229,1,228,1,114,2,113,2,82,1,81,1,46,3,37,8,32,20,32,21,33,58,33,59,34,75,34,76,35,102,35,114,33,120,31,130,30,135,27,145,26,152,29,158,35,158,48,154,54,141,58,128,61,119,67,105,81,103,86,96,94,89,98,81,109,71,119,65,132,60,138,52,151,45,158,40,166,34,172,29,188,26,193,25,200,25,226,24,232,24,270,23,273']), (957285035, 492601069, 445, 29, 591, 24, 419, 0.9923679, [(315, 37, 25), (272, 38, 86), (253, 39, 130), (238, 40, 151), (199, 41, 196), (189, 42, 213), (180, 43, 238), (175, 44, 250), (172, 45, 257), (169, 46, 265), (166, 47, 274), (162, 48, 284), (159, 49, 294), (157, 50, 304), (155, 51, 311), (153, 52, 317), (151, 53, 323), (149, 54, 330), (148, 55, 334), (146, 56, 337), (144, 57, 341), (142, 58, 344), (140, 59, 347), (138, 60, 350), (136, 61, 353), (134, 62, 356), (132, 63, 358), (130, 64, 361), (128, 65, 364), (126, 66, 367), (124, 67, 370), (122, 68, 373), (120, 69, 376), (118, 70, 379), (117, 71, 381), (115, 72, 385), (114, 73, 387), (113, 74, 389), (112, 75, 391), (112, 76, 393), (111, 77, 395), (110, 78, 397), (109, 79, 399), (109, 80, 400), (108, 81, 402), (107, 82, 404), (107, 83, 404), (106, 84, 406), (105, 85, 408), (105, 86, 409), (104, 87, 410), (104, 88, 411), (103, 89, 413), (102, 90, 415), (101, 91, 417), (100, 92, 420), (98, 93, 423), (97, 94, 426), (96, 95, 428), (94, 96, 431), (93, 97, 433), (92, 98, 435), (91, 99, 437), (90, 100, 439), (89, 101, 441), (89, 102, 441), (89, 103, 442), (89, 104, 443), (89, 105, 444), (89, 106, 444), (89, 107, 445), (89, 108, 446), (89, 109, 447), (89, 110, 448), (89, 111, 449), (89, 112, 450), (89, 113, 451), (89, 114, 453), (89, 115, 454), (89, 116, 455), (88, 117, 456), (88, 118, 457), (87, 119, 459), (87, 120, 459), (86, 121, 461), (86, 122, 461), (85, 123, 463), (84, 124, 464), (84, 125, 465), (83, 126, 466), (82, 127, 468), (82, 128, 468), (81, 129, 470), (80, 130, 471), (78, 131, 473), (77, 132, 475), (75, 133, 477), (73, 134, 480), (71, 135, 482), (70, 136, 484), (68, 137, 486), (67, 138, 488), (65, 139, 490), (64, 140, 492), (63, 141, 493), (61, 142, 496), (60, 143, 497), (59, 144, 499), (58, 145, 501), (58, 146, 501), (57, 147, 503), (57, 148, 504), (57, 149, 505), (56, 150, 507), (56, 151, 507), (55, 152, 509), (55, 153, 510), (54, 154, 511), (54, 155, 512), (54, 156, 513), (53, 157, 514), (53, 158, 514), (52, 159, 515), (52, 160, 516), (52, 161, 516), (51, 162, 517), (51, 163, 517), (50, 164, 518), (50, 165, 518), (49, 166, 519), (49, 167, 520), (48, 168, 521), (48, 169, 521), (47, 170, 522), (47, 171, 522), (46, 172, 523), (46, 173, 523), (46, 174, 523), (45, 175, 524), (45, 176, 523), (44, 177, 524), (44, 178, 524), (44, 179, 524), (43, 180, 525), (43, 181, 525), (42, 182, 525), (42, 183, 525), (42, 184, 525), (41, 185, 526), (41, 186, 526), (40, 187, 526), (39, 188, 526), (39, 189, 525), (38, 190, 526), (38, 191, 525), (37, 192, 525), (37, 193, 523), (36, 194, 523), (36, 195, 523), (36, 196, 522), (35, 197, 522), (35, 198, 521), (34, 199, 521), (34, 200, 521), (34, 201, 520), (34, 202, 520), (34, 203, 520), (34, 204, 519), (34, 205, 519), (33, 206, 520), (33, 207, 519), (33, 208, 519), (33, 209, 519), (33, 210, 518), (33, 211, 518), (33, 212, 518), (33, 213, 517), (32, 214, 518), (32, 215, 517), (32, 216, 517), (32, 217, 516), (32, 218, 515), (32, 219, 514), (32, 220, 513), (32, 221, 512), (32, 222, 511), (32, 223, 510), (32, 224, 508), (32, 225, 507), (32, 226, 505), (32, 227, 504), (32, 228, 503), (32, 229, 502), (32, 230, 502), (32, 231, 501), (32, 232, 500), (32, 233, 499), (32, 234, 498), (32, 235, 497), (31, 236, 496), (31, 237, 495), (31, 238, 494), (31, 239, 493), (31, 240, 491), (31, 241, 490), (31, 242, 488), (31, 243, 487), (31, 244, 486), (31, 245, 485), (31, 246, 483), (31, 247, 482), (31, 248, 480), (31, 249, 479), (31, 250, 477), (31, 251, 475), (31, 252, 474), (31, 253, 472), (31, 254, 470), (31, 255, 468), (31, 256, 467), (31, 257, 465), (31, 258, 464), (31, 259, 463), (31, 260, 462), (31, 261, 461), (31, 262, 459), (31, 263, 458), (31, 264, 456), (31, 265, 455), (31, 266, 453), (31, 267, 451), (31, 268, 449), (31, 269, 448), (31, 270, 447), (31, 271, 445), (31, 272, 444), (31, 273, 443), (32, 274, 441), (32, 275, 440), (32, 276, 438), (32, 277, 437), (32, 278, 435), (32, 279, 434), (32, 280, 432), (33, 281, 429), (33, 282, 427), (33, 283, 426), (33, 284, 424), (33, 285, 423), (34, 286, 421), (34, 287, 420), (34, 288, 419), (35, 289, 416), (35, 290, 415), (35, 291, 414), (36, 292, 411), (36, 293, 410), (37, 294, 407), (37, 295, 406), (38, 296, 403), (38, 297, 401), (39, 298, 399), (39, 299, 397), (41, 300, 394), (42, 301, 392), (43, 302, 389), (44, 303, 387), (45, 304, 385), (46, 305, 382), (47, 306, 380), (47, 307, 378), (48, 308, 376), (49, 309, 373), (50, 310, 370), (51, 311, 368), (51, 312, 367), (52, 313, 365), (54, 314, 362), (55, 315, 360), (56, 316, 359), (58, 317, 356), (61, 318, 352), (64, 319, 349), (67, 320, 345), (70, 321, 341), (73, 322, 338), (75, 323, 335), (78, 324, 332), (80, 325, 329), (82, 326, 327), (84, 327, 324), (86, 328, 322), (88, 329, 320), (90, 330, 317), (93, 331, 314), (96, 332, 311), (99, 333, 307), (102, 334, 304), (105, 335, 300), (108, 336, 297), (111, 337, 294), (113, 338, 291), (115, 339, 289), (117, 340, 286), (119, 341, 283), (121, 342, 281), (123, 343, 278), (125, 344, 275), (127, 345, 272), (129, 346, 269), (132, 347, 266), (135, 348, 262), (138, 349, 258), (141, 350, 255), (143, 351, 252), (146, 352, 249), (147, 353, 247), (149, 354, 245), (151, 355, 242), (152, 356, 241), (154, 357, 239), (156, 358, 237), (159, 359, 233), (161, 360, 231), (163, 361, 229), (165, 362, 227), (167, 363, 224), (169, 364, 222), (170, 365, 221), (172, 366, 219), (173, 367, 218), (174, 368, 216), (175, 369, 215), (177, 370, 213), (178, 371, 212), (180, 372, 209), (183, 373, 206), (185, 374, 204), (188, 375, 200), (191, 376, 197), (194, 377, 193), (196, 378, 191), (199, 379, 188), (201, 380, 185), (203, 381, 183), (205, 382, 180), (207, 383, 178), (208, 384, 176), (210, 385, 174), (212, 386, 171), (213, 387, 169), (215, 388, 166), (218, 389, 162), (221, 390, 158), (225, 391, 153), (228, 392, 149), (232, 393, 144), (235, 394, 140), (238, 395, 136), (241, 396, 133), (245, 397, 128), (248, 398, 124), (252, 399, 119), (257, 400, 113), (263, 401, 105), (272, 402, 94), (283, 403, 82), (296, 404, 66), (306, 405, 53), (313, 406, 38), (321, 407, 23)], ['321,407,305,404,263,401,215,388,178,371,168,363,145,351,129,346,110,336,90,330,72,321,56,316,39,299,31,273,31,236,34,199,42,184,58,145,82,128,89,116,89,101,104,88,115,72,159,49,180,43,199,41,237,41,272,38,339,37,382,39,402,43,417,43,460,50,481,55,543,116,556,143,566,156,568,167,566,186,554,199,548,216,515,245,471,275,460,281,414,315,403,339,392,355,383,385,369,400,358,405']), (957285035, 492601069, 445, 485, 636, 23, 174, 0.97116494, [(540, 24, 21), (626, 24, 3), (531, 25, 49), (594, 25, 40), (527, 26, 107), (523, 27, 111), (520, 28, 114), (517, 29, 118), (516, 30, 119), (515, 31, 120), (513, 32, 122), (512, 33, 123), (510, 34, 125), (509, 35, 126), (507, 36, 128), (506, 37, 129), (504, 38, 131), (503, 39, 132), (501, 40, 134), (500, 41, 135), (499, 42, 136), (498, 43, 137), (497, 44, 138), (496, 45, 139), (496, 46, 139), (495, 47, 140), (495, 48, 140), (494, 49, 141), (493, 50, 142), (492, 51, 143), (491, 52, 144), (491, 53, 144), (490, 54, 145), (490, 55, 145), (490, 56, 145), (490, 57, 146), (490, 58, 146), (490, 59, 146), (491, 60, 145), (491, 61, 145), (491, 62, 145), (492, 63, 144), (493, 64, 143), (494, 65, 142), (495, 66, 141), (496, 67, 140), (497, 68, 138), (498, 69, 138), (499, 70, 137), (500, 71, 136), (501, 72, 135), (503, 73, 133), (503, 74, 133), (505, 75, 131), (506, 76, 130), (507, 77, 129), (508, 78, 128), (509, 79, 127), (510, 80, 126), (511, 81, 125), (512, 82, 124), (513, 83, 123), (514, 84, 122), (515, 85, 121), (516, 86, 120), (517, 87, 119), (518, 88, 118), (519, 89, 117), (521, 90, 115), (521, 91, 115), (522, 92, 114), (523, 93, 113), (524, 94, 112), (525, 95, 111), (526, 96, 110), (527, 97, 109), (529, 98, 107), (530, 99, 106), (532, 100, 104), (533, 101, 103), (534, 102, 102), (535, 103, 101), (536, 104, 100), (538, 105, 98), (540, 106, 96), (541, 107, 95), (543, 108, 93), (546, 109, 90), (548, 110, 88), (549, 111, 87), (551, 112, 84), (552, 113, 83), (553, 114, 82), (555, 115, 80), (556, 116, 79), (556, 117, 79), (557, 118, 78), (558, 119, 77), (559, 120, 76), (560, 121, 75), (560, 122, 75), (561, 123, 74), (561, 124, 74), (561, 125, 74), (562, 126, 73), (562, 127, 73), (563, 128, 72), (563, 129, 72), (564, 130, 70), (564, 131, 70), (565, 132, 69), (565, 133, 68), (565, 134, 68), (565, 135, 67), (566, 136, 65), (566, 137, 64), (566, 138, 64), (566, 139, 62), (566, 140, 61), (566, 141, 59), (566, 142, 57), (566, 143, 56), (566, 144, 55), (566, 145, 54), (567, 146, 53), (567, 147, 52), (567, 148, 51), (568, 149, 50), (568, 150, 49), (568, 151, 48), (568, 152, 47), (569, 153, 45), (569, 154, 44), (570, 155, 42), (570, 156, 42), (570, 157, 41), (571, 158, 39), (571, 159, 39), (572, 160, 37), (572, 161, 37), (573, 162, 35), (573, 163, 34), (573, 164, 34), (574, 165, 32), (575, 166, 30), (577, 167, 28), (578, 168, 26), (581, 169, 22), (584, 170, 19), (587, 171, 15), (591, 172, 8)], ['598,172,591,172,590,171,578,168,573,164,573,162,568,152,568,149,566,145,566,136,565,132,561,125,560,121,556,116,547,109,543,108,536,104,531,99,527,97,491,62,490,54,495,48,496,45,501,40,514,32,517,29,531,25,539,25,540,24,560,24,561,25,579,25,580,26,593,26,594,25,633,25,634,29,634,56,635,57,635,111,634,112,634,129,632,134,629,138,623,141,619,145,617,149,611,155,608,161,604,166']), (957285035, 492601069, 445, 280, 481, 2, 55, 0.8298239, [(292, 3, 128), (284, 4, 146), (282, 5, 151), (281, 6, 154), (281, 7, 156), (281, 8, 157), (281, 9, 158), (281, 10, 160), (281, 11, 162), (281, 12, 165), (281, 13, 167), (281, 14, 169), (281, 15, 171), (281, 16, 173), (281, 17, 174), (281, 18, 175), (281, 19, 177), (281, 20, 178), (281, 21, 179), (281, 22, 180), (281, 23, 181), (281, 24, 182), (281, 25, 183), (281, 26, 184), (281, 27, 185), (281, 28, 185), (281, 29, 185), (282, 30, 185), (283, 31, 27), (337, 31, 131), (371, 32, 97), (401, 33, 68), (409, 34, 61), (419, 35, 52), (424, 36, 48), (429, 37, 44), (432, 38, 41), (434, 39, 40), (436, 40, 39), (438, 41, 37), (441, 42, 35), (444, 43, 32), (448, 44, 29), (452, 45, 25), (454, 46, 23), (459, 47, 17), (463, 48, 12), (468, 49, 5)], ['472,49,468,49,467,48,459,47,458,46,454,46,451,44,448,44,447,43,444,43,440,41,438,41,428,36,424,36,423,35,419,35,418,34,409,34,408,33,401,33,400,32,371,32,370,31,337,31,336,30,283,31,281,29,281,6,284,4,291,4,292,3,419,3,420,4,429,4,430,5,432,5,436,7,441,11,445,12,453,16,456,19,457,19,465,27,465,29,472,37,476,44,476,46']), (957285035, 492601069, 445, 456, 547, 6, 45, 0.7401768, [(482, 8, 19), (464, 9, 3), (481, 9, 44), (457, 10, 12), (479, 10, 50), (457, 11, 13), (476, 11, 56), (457, 12, 15), (475, 12, 65), (457, 13, 84), (457, 14, 85), (457, 15, 89), (457, 16, 89), (458, 17, 88), (459, 18, 87), (460, 19, 86), (461, 20, 80), (464, 21, 71), (466, 22, 63), (467, 23, 59), (468, 24, 55), (469, 25, 52), (469, 26, 51), (470, 27, 48), (471, 28, 46), (471, 29, 44), (472, 30, 42), (473, 31, 39), (473, 32, 38), (474, 33, 36), (475, 34, 33), (475, 35, 32), (476, 36, 30), (476, 37, 29), (477, 38, 26), (478, 39, 23), (479, 40, 20), (480, 41, 17), (488, 42, 5)], ['492,42,488,42,487,41,480,41,476,37,475,34,473,32,469,25,465,21,461,20,457,16,457,10,463,10,464,9,466,9,470,12,474,13,476,11,480,10,482,8,500,8,501,9,524,9,525,10,528,10,532,12,539,12,542,15,545,15,545,19,535,20,534,21,529,21,525,23,523,23,513,30,512,30,504,37,496,41,493,41'])], 'temp/1738690527_598972_957285035_a42482e51c93c8025d243dd179aee85b.jpg']} free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3139 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.1328296661376953 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Tue Feb 4 18:35:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3139 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-04 18:35:48.263843: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-04 18:35:48.291158: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-04 18:35:48.293170: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7ccc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-04 18:35:48.293219: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-04 18:35:48.296660: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-04 18:35:48.550211: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d68c220 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-04 18:35:48.550266: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-04 18:35:48.551424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:35:48.555394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:35:48.559176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:35:48.562085: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:35:48.563335: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:35:48.566675: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:35:48.568242: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:35:48.572565: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:35:48.573604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:35:48.573682: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:35:48.574212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 18:35:48.574227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 18:35:48.574236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 18:35:48.575139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2687 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-04 18:35:48.649647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:35:48.649787: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:35:48.649814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:35:48.649839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:35:48.649863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:35:48.649886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:35:48.649908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:35:48.649932: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:35:48.651001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:35:48.652126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:35:48.652170: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:35:48.652191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:35:48.652211: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:35:48.652230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:35:48.652249: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:35:48.652269: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:35:48.652288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:35:48.653302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:35:48.653340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 18:35:48.653351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 18:35:48.653360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 18:35:48.654413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2687 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-04 18:35:55.406029: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:35:55.580574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:35:56.934402: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:56.934489: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:56.935065: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:56.935085: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:56.942075: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:56.942096: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:56.942635: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:56.942649: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:56.949103: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:56.949123: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:56.949673: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:56.949688: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:56.980159: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:56.980185: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:56.980715: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:56.999483: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:57.005260: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.005280: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:57.005820: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.005835: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:35:57.038640: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.039273: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.041041: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.041583: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.084493: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.085069: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.092410: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.092964: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.097472: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.098025: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.109742: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.110294: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.111906: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.112455: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.118502: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.119066: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.120814: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.121358: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.127594: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.128140: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.129757: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.130299: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.159536: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.160101: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.160643: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.161184: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.165002: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.165552: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.183050: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.183649: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.184200: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.184776: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.199601: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.200211: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.200785: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.201330: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.206116: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.206676: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.211689: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.212292: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.226431: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.227018: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.231296: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.231865: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.232717: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.233315: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.254298: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.254879: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.255450: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.256000: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.256547: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.257093: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.271239: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.271788: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.289447: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.289502: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-04 18:35:57.290570: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.291657: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.298620: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.299191: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.307191: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.307734: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.322014: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.322584: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.323160: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.323771: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.327793: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.328332: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.328870: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.329408: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.330311: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.340500: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.341047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.351162: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.351712: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.352261: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.352803: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.353352: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:35:57.353892: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (720, 1280, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 1280.00000 nb d'objets trouves : 4 Detection mask done ! Trying to reset tf kernel 599904 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1946 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3139 list_Values should be empty [] ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0036890506744384766 nb_pixel_total : 16902 time to create 1 rle with old method : 0.03473019599914551 length of segment : 107 time for calcul the mask position with numpy : 0.024788379669189453 nb_pixel_total : 480745 time to create 1 rle with new method : 0.07269430160522461 length of segment : 632 time for calcul the mask position with numpy : 0.0006282329559326172 nb_pixel_total : 36584 time to create 1 rle with old method : 0.041281700134277344 length of segment : 132 time for calcul the mask position with numpy : 0.00010752677917480469 nb_pixel_total : 4794 time to create 1 rle with old method : 0.00581812858581543 length of segment : 51 time spent for convertir_results : 0.3985271453857422 time spend for datou_step_exec : 15.40071964263916 time spend to save output : 3.266334533691406e-05 total time spend for step 1 : 15.400752305984497 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 395 chid ids of type : 445 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.010593891143798828 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'917855882': [[(917855882, 492601069, 445, 1092, 1280, 0, 108, 0.9988386, [(1205, 1, 58), (1165, 2, 105), (1159, 3, 113), (1149, 4, 124), (1113, 5, 161), (1100, 6, 174), (1097, 7, 177), (1095, 8, 179), (1095, 9, 179), (1095, 10, 179), (1095, 11, 179), (1095, 12, 179), (1095, 13, 179), (1095, 14, 178), (1095, 15, 178), (1095, 16, 178), (1095, 17, 178), (1095, 18, 177), (1095, 19, 177), (1095, 20, 177), (1095, 21, 177), (1095, 22, 177), (1095, 23, 178), (1095, 24, 178), (1095, 25, 178), (1095, 26, 179), (1095, 27, 179), (1095, 28, 180), (1095, 29, 181), (1095, 30, 182), (1095, 31, 183), (1095, 32, 183), (1095, 33, 184), (1095, 34, 184), (1096, 35, 183), (1096, 36, 183), (1096, 37, 184), (1097, 38, 183), (1097, 39, 183), (1097, 40, 183), (1098, 41, 182), (1098, 42, 182), (1098, 43, 182), (1099, 44, 181), (1099, 45, 181), (1099, 46, 181), (1100, 47, 180), (1100, 48, 180), (1101, 49, 179), (1101, 50, 179), (1102, 51, 178), (1102, 52, 178), (1103, 53, 177), (1103, 54, 177), (1104, 55, 176), (1104, 56, 176), (1104, 57, 176), (1104, 58, 176), (1105, 59, 175), (1105, 60, 175), (1105, 61, 175), (1105, 62, 175), (1105, 63, 175), (1106, 64, 174), (1106, 65, 174), (1106, 66, 174), (1106, 67, 174), (1106, 68, 174), (1106, 69, 174), (1106, 70, 174), (1106, 71, 174), (1106, 72, 174), (1106, 73, 174), (1107, 74, 173), (1107, 75, 173), (1107, 76, 173), (1107, 77, 173), (1107, 78, 173), (1107, 79, 173), (1108, 80, 172), (1108, 81, 172), (1109, 82, 171), (1110, 83, 170), (1110, 84, 170), (1111, 85, 169), (1112, 86, 168), (1113, 87, 166), (1114, 88, 165), (1115, 89, 164), (1117, 90, 162), (1120, 91, 159), (1138, 92, 141), (1146, 93, 133), (1154, 94, 125), (1167, 95, 112), (1177, 96, 102), (1183, 97, 95), (1185, 98, 93), (1187, 99, 90), (1188, 100, 55), (1264, 100, 12), (1190, 101, 50), (1191, 102, 46), (1194, 103, 40), (1197, 104, 34), (1202, 105, 25), (1207, 106, 16)], ['1222,106,1207,106,1206,105,1197,104,1191,102,1182,96,1176,95,1167,95,1166,94,1154,94,1153,93,1146,93,1145,92,1137,91,1120,91,1115,89,1110,84,1107,79,1106,73,1106,64,1104,55,1099,46,1095,34,1095,8,1100,6,1112,6,1113,5,1148,5,1149,4,1158,4,1165,2,1204,2,1205,1,1262,1,1269,2,1273,5,1273,13,1271,18,1271,22,1273,27,1277,31,1279,37,1279,86,1278,87,1278,96,1275,100,1264,100,1263,99,1243,99,1230,104']), (917855882, 492601069, 445, 52, 1128, 16, 668, 0.9977418, [(711, 22, 21), (926, 22, 46), (608, 23, 146), (894, 23, 103), (598, 24, 233), (850, 24, 158), (590, 25, 427), (582, 26, 444), (575, 27, 458), (569, 28, 466), (565, 29, 472), (560, 30, 480), (556, 31, 486), (550, 32, 495), (544, 33, 503), (538, 34, 512), (532, 35, 520), (527, 36, 527), (523, 37, 534), (518, 38, 541), (514, 39, 548), (510, 40, 554), (506, 41, 561), (503, 42, 566), (499, 43, 572), (496, 44, 577), (493, 45, 582), (491, 46, 585), (489, 47, 589), (487, 48, 592), (485, 49, 595), (483, 50, 598), (482, 51, 600), (481, 52, 602), (480, 53, 603), (479, 54, 605), (478, 55, 606), (476, 56, 608), (475, 57, 610), (474, 58, 611), (473, 59, 613), (472, 60, 614), (470, 61, 616), (469, 62, 618), (468, 63, 619), (466, 64, 621), (465, 65, 623), (464, 66, 624), (462, 67, 626), (461, 68, 628), (459, 69, 630), (458, 70, 631), (456, 71, 633), (455, 72, 635), (453, 73, 637), (452, 74, 638), (451, 75, 639), (450, 76, 640), (448, 77, 642), (447, 78, 643), (446, 79, 644), (445, 80, 645), (444, 81, 646), (442, 82, 648), (441, 83, 649), (440, 84, 650), (439, 85, 651), (438, 86, 652), (437, 87, 653), (436, 88, 654), (435, 89, 655), (434, 90, 656), (433, 91, 657), (432, 92, 658), (431, 93, 659), (430, 94, 660), (429, 95, 661), (428, 96, 662), (427, 97, 663), (425, 98, 665), (423, 99, 667), (421, 100, 669), (419, 101, 671), (417, 102, 673), (413, 103, 677), (410, 104, 680), (405, 105, 685), (401, 106, 689), (397, 107, 693), (392, 108, 698), (387, 109, 703), (382, 110, 708), (377, 111, 713), (373, 112, 717), (368, 113, 722), (365, 114, 725), (361, 115, 729), (358, 116, 732), (356, 117, 734), (353, 118, 737), (351, 119, 739), (348, 120, 742), (346, 121, 744), (344, 122, 746), (341, 123, 749), (338, 124, 752), (335, 125, 755), (331, 126, 759), (327, 127, 763), (323, 128, 767), (319, 129, 770), (314, 130, 775), (308, 131, 781), (303, 132, 786), (294, 133, 795), (286, 134, 803), (279, 135, 810), (273, 136, 816), (266, 137, 823), (262, 138, 827), (258, 139, 831), (255, 140, 834), (252, 141, 837), (250, 142, 839), (247, 143, 842), (245, 144, 844), (242, 145, 847), (240, 146, 849), (237, 147, 852), (233, 148, 856), (230, 149, 859), (226, 150, 863), (220, 151, 869), (213, 152, 876), (206, 153, 883), (200, 154, 889), (193, 155, 896), (187, 156, 902), (183, 157, 906), (181, 158, 908), (178, 159, 911), (176, 160, 913), (174, 161, 915), (172, 162, 917), (170, 163, 919), (168, 164, 921), (166, 165, 923), (165, 166, 924), (164, 167, 925), (162, 168, 927), (161, 169, 928), (159, 170, 930), (157, 171, 932), (155, 172, 934), (153, 173, 935), (151, 174, 937), (148, 175, 940), (146, 176, 942), (144, 177, 944), (142, 178, 946), (140, 179, 948), (139, 180, 949), (137, 181, 951), (136, 182, 952), (134, 183, 954), (133, 184, 955), (132, 185, 956), (131, 186, 957), (130, 187, 958), (129, 188, 959), (128, 189, 960), (127, 190, 960), (126, 191, 961), (126, 192, 961), (125, 193, 962), (124, 194, 963), (123, 195, 964), (122, 196, 965), (122, 197, 965), (121, 198, 966), (120, 199, 967), (119, 200, 968), (118, 201, 969), (117, 202, 970), (116, 203, 971), (114, 204, 973), (113, 205, 973), (112, 206, 974), (111, 207, 975), (109, 208, 977), (108, 209, 978), (107, 210, 979), (106, 211, 980), (105, 212, 981), (104, 213, 982), (103, 214, 983), (102, 215, 984), (101, 216, 985), (101, 217, 984), (100, 218, 985), (99, 219, 986), (99, 220, 986), (98, 221, 987), (98, 222, 987), (97, 223, 988), (97, 224, 987), (96, 225, 988), (96, 226, 988), (95, 227, 989), (95, 228, 989), (94, 229, 990), (94, 230, 990), (94, 231, 990), (93, 232, 990), (93, 233, 990), (92, 234, 991), (92, 235, 991), (92, 236, 991), (91, 237, 992), (91, 238, 991), (91, 239, 991), (91, 240, 990), (91, 241, 990), (90, 242, 991), (90, 243, 990), (90, 244, 990), (90, 245, 989), (90, 246, 989), (89, 247, 990), (89, 248, 989), (89, 249, 989), (89, 250, 988), (89, 251, 988), (88, 252, 988), (88, 253, 988), (88, 254, 987), (88, 255, 986), (88, 256, 986), (87, 257, 986), (87, 258, 985), (87, 259, 985), (87, 260, 984), (87, 261, 983), (86, 262, 983), (86, 263, 982), (86, 264, 982), (86, 265, 981), (85, 266, 981), (85, 267, 980), (85, 268, 980), (84, 269, 980), (84, 270, 979), (84, 271, 979), (84, 272, 978), (83, 273, 979), (83, 274, 978), (83, 275, 977), (82, 276, 978), (82, 277, 977), (82, 278, 977), (81, 279, 977), (81, 280, 977), (81, 281, 977), (80, 282, 977), (80, 283, 977), (80, 284, 976), (79, 285, 977), (79, 286, 976), (79, 287, 976), (78, 288, 976), (78, 289, 976), (78, 290, 975), (77, 291, 976), (77, 292, 975), (77, 293, 975), (76, 294, 975), (76, 295, 975), (76, 296, 974), (75, 297, 975), (75, 298, 974), (74, 299, 975), (74, 300, 974), (74, 301, 974), (73, 302, 974), (73, 303, 974), (72, 304, 974), (72, 305, 974), (71, 306, 974), (71, 307, 973), (71, 308, 972), (70, 309, 972), (70, 310, 971), (70, 311, 970), (70, 312, 968), (69, 313, 968), (69, 314, 966), (69, 315, 964), (69, 316, 962), (68, 317, 961), (68, 318, 959), (68, 319, 958), (68, 320, 956), (67, 321, 955), (67, 322, 954), (67, 323, 952), (67, 324, 951), (66, 325, 951), (66, 326, 950), (66, 327, 948), (66, 328, 947), (65, 329, 947), (65, 330, 946), (65, 331, 946), (65, 332, 945), (65, 333, 944), (65, 334, 942), (65, 335, 941), (65, 336, 940), (65, 337, 939), (65, 338, 938), (64, 339, 937), (64, 340, 936), (64, 341, 934), (64, 342, 932), (64, 343, 930), (64, 344, 928), (64, 345, 926), (64, 346, 925), (64, 347, 923), (64, 348, 922), (64, 349, 920), (64, 350, 919), (63, 351, 919), (63, 352, 918), (63, 353, 917), (63, 354, 916), (63, 355, 915), (63, 356, 914), (63, 357, 912), (63, 358, 911), (63, 359, 910), (63, 360, 909), (63, 361, 908), (63, 362, 906), (63, 363, 905), (63, 364, 904), (63, 365, 902), (63, 366, 901), (63, 367, 899), (63, 368, 898), (62, 369, 897), (62, 370, 895), (62, 371, 893), (62, 372, 891), (62, 373, 890), (62, 374, 888), (62, 375, 887), (62, 376, 886), (62, 377, 885), (62, 378, 884), (62, 379, 883), (63, 380, 880), (63, 381, 879), (63, 382, 878), (63, 383, 877), (63, 384, 876), (63, 385, 875), (63, 386, 874), (63, 387, 873), (63, 388, 872), (64, 389, 870), (64, 390, 869), (64, 391, 868), (64, 392, 867), (64, 393, 865), (64, 394, 864), (64, 395, 863), (65, 396, 861), (65, 397, 860), (65, 398, 859), (65, 399, 858), (65, 400, 857), (65, 401, 856), (65, 402, 854), (65, 403, 853), (65, 404, 851), (65, 405, 850), (65, 406, 848), (66, 407, 846), (66, 408, 844), (66, 409, 843), (66, 410, 842), (66, 411, 841), (66, 412, 840), (66, 413, 838), (66, 414, 837), (66, 415, 836), (66, 416, 835), (66, 417, 835), (66, 418, 834), (66, 419, 833), (67, 420, 831), (67, 421, 830), (67, 422, 829), (67, 423, 829), (67, 424, 828), (67, 425, 827), (67, 426, 826), (67, 427, 825), (67, 428, 824), (68, 429, 822), (68, 430, 820), (68, 431, 819), (68, 432, 818), (68, 433, 816), (68, 434, 815), (68, 435, 813), (68, 436, 811), (69, 437, 809), (69, 438, 807), (69, 439, 805), (69, 440, 804), (69, 441, 803), (69, 442, 802), (69, 443, 800), (70, 444, 798), (70, 445, 797), (70, 446, 796), (70, 447, 796), (71, 448, 794), (71, 449, 794), (72, 450, 792), (72, 451, 791), (73, 452, 790), (73, 453, 789), (74, 454, 788), (74, 455, 787), (75, 456, 786), (75, 457, 785), (76, 458, 784), (76, 459, 783), (77, 460, 782), (77, 461, 781), (77, 462, 781), (78, 463, 779), (78, 464, 779), (79, 465, 777), (79, 466, 777), (79, 467, 776), (80, 468, 775), (80, 469, 774), (80, 470, 774), (81, 471, 772), (81, 472, 771), (82, 473, 770), (82, 474, 769), (83, 475, 767), (83, 476, 766), (83, 477, 766), (84, 478, 764), (84, 479, 763), (85, 480, 761), (85, 481, 760), (85, 482, 759), (86, 483, 757), (86, 484, 755), (87, 485, 753), (87, 486, 752), (87, 487, 750), (88, 488, 748), (88, 489, 747), (88, 490, 746), (89, 491, 744), (89, 492, 743), (90, 493, 741), (90, 494, 741), (91, 495, 739), (91, 496, 738), (92, 497, 736), (93, 498, 735), (94, 499, 733), (94, 500, 733), (95, 501, 731), (96, 502, 729), (97, 503, 728), (98, 504, 726), (99, 505, 724), (99, 506, 724), (100, 507, 722), (101, 508, 721), (102, 509, 719), (104, 510, 717), (105, 511, 715), (106, 512, 714), (107, 513, 712), (108, 514, 711), (110, 515, 708), (111, 516, 707), (113, 517, 704), (114, 518, 703), (115, 519, 701), (117, 520, 698), (118, 521, 697), (119, 522, 695), (121, 523, 693), (122, 524, 691), (124, 525, 689), (125, 526, 687), (126, 527, 685), (128, 528, 683), (129, 529, 681), (131, 530, 678), (132, 531, 676), (134, 532, 673), (135, 533, 672), (137, 534, 669), (138, 535, 667), (140, 536, 664), (141, 537, 662), (143, 538, 659), (144, 539, 657), (146, 540, 654), (148, 541, 651), (149, 542, 649), (151, 543, 645), (153, 544, 642), (154, 545, 640), (156, 546, 638), (158, 547, 635), (159, 548, 633), (161, 549, 630), (162, 550, 628), (164, 551, 626), (165, 552, 624), (167, 553, 621), (168, 554, 619), (170, 555, 617), (171, 556, 615), (173, 557, 613), (174, 558, 611), (176, 559, 608), (177, 560, 607), (178, 561, 605), (180, 562, 603), (181, 563, 601), (183, 564, 599), (184, 565, 598), (186, 566, 595), (189, 567, 592), (192, 568, 589), (195, 569, 585), (198, 570, 582), (201, 571, 579), (204, 572, 575), (206, 573, 573), (209, 574, 569), (212, 575, 566), (215, 576, 563), (218, 577, 559), (221, 578, 556), (223, 579, 553), (226, 580, 550), (228, 581, 547), (230, 582, 545), (232, 583, 542), (234, 584, 540), (235, 585, 539), (237, 586, 536), (238, 587, 534), (240, 588, 531), (242, 589, 528), (243, 590, 526), (245, 591, 523), (247, 592, 520), (249, 593, 516), (251, 594, 513), (253, 595, 510), (256, 596, 505), (258, 597, 501), (261, 598, 497), (263, 599, 493), (267, 600, 488), (271, 601, 482), (274, 602, 478), (278, 603, 473), (281, 604, 468), (284, 605, 464), (287, 606, 460), (290, 607, 456), (292, 608, 453), (295, 609, 449), (298, 610, 445), (300, 611, 442), (303, 612, 438), (305, 613, 434), (308, 614, 430), (310, 615, 427), (312, 616, 423), (315, 617, 418), (317, 618, 415), (320, 619, 410), (322, 620, 406), (325, 621, 401), (328, 622, 395), (330, 623, 390), (333, 624, 384), (335, 625, 379), (338, 626, 374), (341, 627, 369), (345, 628, 362), (349, 629, 356), (353, 630, 350), (357, 631, 344), (360, 632, 340), (364, 633, 334), (368, 634, 327), (373, 635, 320), (378, 636, 313), (383, 637, 305), (389, 638, 294), (395, 639, 282), (401, 640, 270), (408, 641, 256), (416, 642, 240), (432, 643, 216), (448, 644, 193), (465, 645, 169), (480, 646, 148), (495, 647, 126), (511, 648, 104), (526, 649, 81), (565, 650, 9)], ['526,649,416,642,341,627,243,590,220,577,186,566,102,509,91,496,70,447,62,379,65,329,86,265,91,237,101,216,134,183,187,156,225,151,266,137,302,133,343,123,358,116,416,103,483,50,506,41,569,28,608,23,754,24,893,24,926,22,996,23,1032,27,1066,41,1082,52,1089,72,1088,172,1082,237,1045,305,1019,322,1002,338,950,373,889,429,865,446,851,473,830,493,810,528,786,554,773,585,740,612,687,637,606,649']), (917855882, 492601069, 445, 0, 438, 0, 116, 0.99194664, [(127, 1, 140), (94, 2, 205), (59, 3, 273), (338, 3, 59), (22, 4, 380), (19, 5, 386), (16, 6, 391), (15, 7, 393), (14, 8, 395), (14, 9, 396), (13, 10, 398), (12, 11, 399), (12, 12, 399), (11, 13, 401), (10, 14, 402), (11, 15, 402), (11, 16, 403), (12, 17, 403), (12, 18, 404), (12, 19, 405), (12, 20, 405), (12, 21, 405), (12, 22, 406), (12, 23, 406), (12, 24, 406), (12, 25, 407), (12, 26, 407), (12, 27, 407), (12, 28, 408), (12, 29, 408), (12, 30, 408), (12, 31, 408), (12, 32, 408), (12, 33, 408), (12, 34, 409), (12, 35, 409), (12, 36, 409), (12, 37, 409), (12, 38, 409), (12, 39, 409), (12, 40, 409), (12, 41, 409), (12, 42, 409), (12, 43, 410), (12, 44, 410), (12, 45, 409), (12, 46, 409), (12, 47, 409), (12, 48, 409), (12, 49, 409), (12, 50, 409), (12, 51, 409), (12, 52, 408), (12, 53, 408), (12, 54, 407), (12, 55, 406), (12, 56, 405), (12, 57, 404), (12, 58, 402), (11, 59, 402), (11, 60, 401), (11, 61, 401), (11, 62, 400), (11, 63, 399), (11, 64, 399), (11, 65, 398), (11, 66, 397), (11, 67, 396), (11, 68, 395), (11, 69, 395), (11, 70, 394), (11, 71, 394), (11, 72, 394), (11, 73, 393), (11, 74, 393), (11, 75, 393), (11, 76, 393), (11, 77, 393), (11, 78, 393), (11, 79, 392), (11, 80, 392), (10, 81, 393), (10, 82, 394), (10, 83, 394), (9, 84, 396), (9, 85, 263), (284, 85, 121), (9, 86, 75), (96, 86, 30), (141, 86, 120), (292, 86, 112), (9, 87, 71), (151, 87, 105), (294, 87, 110), (8, 88, 67), (160, 88, 92), (295, 88, 108), (8, 89, 63), (175, 89, 74), (296, 89, 107), (7, 90, 61), (204, 90, 41), (297, 90, 105), (7, 91, 57), (298, 91, 104), (7, 92, 53), (299, 92, 103), (6, 93, 50), (300, 93, 101), (7, 94, 46), (303, 94, 96), (7, 95, 44), (305, 95, 93), (7, 96, 42), (308, 96, 88), (7, 97, 40), (310, 97, 85), (7, 98, 38), (312, 98, 82), (8, 99, 34), (314, 99, 79), (8, 100, 32), (316, 100, 75), (8, 101, 29), (319, 101, 71), (13, 102, 18), (325, 102, 62), (20, 103, 6), (331, 103, 51), (337, 104, 37), (344, 105, 22), (351, 106, 2)], ['344,105,319,101,307,95,300,93,283,84,261,85,244,90,204,90,203,89,175,89,160,88,140,85,125,86,83,86,56,92,36,101,25,103,8,101,6,93,11,80,11,59,12,58,12,17,10,14,16,6,22,4,58,4,59,3,93,3,94,2,126,2,127,1,266,1,267,2,298,2,299,3,396,3,406,6,416,19,420,34,420,51,411,60,405,68,402,81,404,85,401,92,392,99,386,102,374,103,365,105']), (917855882, 492601069, 445, 390, 550, 0, 54, 0.9391709, [(414, 0, 7), (441, 0, 60), (508, 0, 28), (402, 1, 142), (401, 2, 146), (402, 3, 145), (404, 4, 143), (406, 5, 140), (408, 6, 137), (410, 7, 134), (411, 8, 132), (412, 9, 130), (413, 10, 127), (414, 11, 125), (415, 12, 123), (415, 13, 122), (416, 14, 120), (417, 15, 117), (417, 16, 116), (418, 17, 114), (418, 18, 113), (418, 19, 111), (418, 20, 109), (419, 21, 107), (419, 22, 105), (419, 23, 103), (419, 24, 102), (419, 25, 100), (420, 26, 97), (420, 27, 95), (420, 28, 94), (421, 29, 91), (421, 30, 90), (422, 31, 88), (422, 32, 88), (422, 33, 87), (423, 34, 84), (423, 35, 82), (423, 36, 81), (424, 37, 79), (424, 38, 77), (424, 39, 75), (424, 40, 73), (424, 41, 71), (425, 42, 67), (425, 43, 66), (426, 44, 62), (426, 45, 6), (433, 45, 52), (443, 46, 30), (450, 47, 1)], ['450,47,449,46,443,46,442,45,426,45,424,41,424,37,423,36,422,31,419,25,419,21,418,20,418,17,417,15,409,6,402,3,402,1,413,1,414,0,420,0,421,1,440,1,441,0,500,0,501,1,507,1,508,0,535,0,536,1,543,1,546,2,546,4,542,8,530,18,527,19,525,21,522,22,520,24,512,28,508,33,505,34,502,37,494,41,492,41,490,43,488,43,484,45,473,45,472,46,451,46'])], 'temp/1738690545_598972_917855882_da0fa7b7e6b5b551fe26c0ba8713276d.jpg']} ############################### TEST POLYGON ################################ Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.10047364234924316 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Tue Feb 4 18:36:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2788 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-04 18:36:05.039325: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-04 18:36:05.067322: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-04 18:36:05.069625: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7cd4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-04 18:36:05.069679: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-04 18:36:05.073585: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-04 18:36:05.293785: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d8dcb20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-04 18:36:05.293836: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-04 18:36:05.294605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:36:05.295032: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:36:05.297505: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:36:05.299464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:36:05.299940: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:36:05.302724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:36:05.304623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:36:05.311576: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:36:05.313114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:36:05.313240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:36:05.313948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 18:36:05.313972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 18:36:05.313984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 18:36:05.315293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1888 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-04 18:36:05.409477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:36:05.409566: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:36:05.409599: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:36:05.409616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:36:05.409633: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:36:05.409649: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:36:05.409665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:36:05.409681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:36:05.410560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:36:05.411511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 18:36:05.411545: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 18:36:05.411562: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:36:05.411578: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 18:36:05.411594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 18:36:05.411609: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 18:36:05.411625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 18:36:05.411641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:36:05.412457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 18:36:05.412492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 18:36:05.412501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 18:36:05.412508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 18:36:05.413372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1888 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-04 18:36:09.765828: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 51380224 exceeds 10% of free system memory. 2025-02-04 18:36:14.675372: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 18:36:14.940101: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 18:36:16.558151: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.610406: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.616710: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.616750: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.662314: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.662395: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.697802: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.697830: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.780264: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.780296: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-04 18:36:16.781635: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-02-04 18:36:16.795741: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.796508: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.21G (1300326144 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.797361: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.09G (1170293504 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.798240: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.813836: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.814322: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.830090: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.830573: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.840581: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.841067: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.842479: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.842970: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.848147: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.848640: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.850183: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.850664: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.856077: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.856567: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.857983: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.858464: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.884058: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.884553: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.885037: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.885520: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.888853: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.889336: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.904345: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.904838: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.905323: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.905804: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.917869: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.918389: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.918874: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.919409: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.923655: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.924144: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.928603: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.929141: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.941140: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.941658: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.945845: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.946395: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.967569: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.968085: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.968624: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.969142: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.969662: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:16.970160: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.050976: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.051066: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-04 18:36:17.052216: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.053091: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.061757: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.062593: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.083953: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.084528: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.100459: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.101063: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.101605: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.102152: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.106378: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.107009: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.107653: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.108575: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.110300: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.120220: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.120812: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.131395: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.131999: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.132543: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.133070: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.133608: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-04 18:36:17.134135: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.34G (1444806656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (2448, 2448, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 2448.00000 nb d'objets trouves : 1 Detection mask done ! Trying to reset tf kernel 600718 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 753 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3139 list_Values should be empty [] ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 1.5848181247711182 nb_pixel_total : 3706912 time to create 1 rle with new method : 0.8961107730865479 length of segment : 1991 time spent for convertir_results : 3.5544071197509766 time spend for datou_step_exec : 21.768729209899902 time spend to save output : 3.337860107421875e-05 total time spend for step 1 : 21.768762588500977 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 717 chid ids of type : 445 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 1991 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.011875391006469727 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'917877156': [[(917877156, 492601069, 445, 0, 2371, 99, 2103, 0.9919618, [(705, 108, 130), (1287, 108, 32), (671, 109, 324), (1135, 109, 201), (639, 110, 413), (1066, 110, 287), (546, 111, 823), (539, 112, 846), (531, 113, 874), (524, 114, 926), (518, 115, 965), (511, 116, 977), (505, 117, 988), (499, 118, 999), (493, 119, 1009), (487, 120, 1019), (482, 121, 1029), (477, 122, 1038), (472, 123, 1046), (467, 124, 1055), (462, 125, 1064), (457, 126, 1072), (453, 127, 1080), (449, 128, 1087), (445, 129, 1094), (440, 130, 1102), (437, 131, 1108), (433, 132, 1115), (429, 133, 1122), (425, 134, 1129), (422, 135, 1135), (418, 136, 1141), (415, 137, 1147), (411, 138, 1154), (407, 139, 1162), (403, 140, 1171), (399, 141, 1180), (395, 142, 1190), (391, 143, 1199), (387, 144, 1209), (382, 145, 1220), (379, 146, 1229), (376, 147, 1238), (373, 148, 1247), (370, 149, 1257), (367, 150, 1267), (364, 151, 1277), (361, 152, 1287), (358, 153, 1297), (355, 154, 1307), (351, 155, 1318), (348, 156, 1328), (344, 157, 1339), (341, 158, 1349), (337, 159, 1361), (333, 160, 1373), (329, 161, 1386), (325, 162, 1399), (321, 163, 1412), (317, 164, 1425), (313, 165, 1438), (309, 166, 1452), (304, 167, 1466), (300, 168, 1480), (295, 169, 1495), (292, 170, 1508), (288, 171, 1522), (285, 172, 1535), (282, 173, 1542), (279, 174, 1548), (276, 175, 1554), (273, 176, 1560), (270, 177, 1566), (268, 178, 1571), (265, 179, 1576), (262, 180, 1582), (260, 181, 1586), (257, 182, 1592), (255, 183, 1596), (252, 184, 1601), (250, 185, 1605), (248, 186, 1609), (246, 187, 1613), (244, 188, 1617), (242, 189, 1621), (239, 190, 1625), (237, 191, 1629), (236, 192, 1632), (234, 193, 1635), (232, 194, 1639), (230, 195, 1642), (228, 196, 1646), (226, 197, 1649), (225, 198, 1652), (223, 199, 1655), (221, 200, 1658), (220, 201, 1661), (218, 202, 1664), (217, 203, 1666), (215, 204, 1669), (214, 205, 1671), (212, 206, 1674), (211, 207, 1676), (209, 208, 1679), (208, 209, 1682), (206, 210, 1685), (205, 211, 1687), (204, 212, 1689), (202, 213, 1692), (201, 214, 1695), (199, 215, 1698), (198, 216, 1700), (196, 217, 1704), (195, 218, 1706), (193, 219, 1709), (192, 220, 1712), (190, 221, 1716), (189, 222, 1718), (187, 223, 1722), (185, 224, 1726), (184, 225, 1729), (182, 226, 1732), (181, 227, 1735), (179, 228, 1739), (178, 229, 1742), (176, 230, 1745), (175, 231, 1748), (173, 232, 1752), (171, 233, 1755), (170, 234, 1758), (168, 235, 1761), (166, 236, 1765), (165, 237, 1768), (163, 238, 1771), (161, 239, 1775), (160, 240, 1777), (158, 241, 1781), (156, 242, 1784), (155, 243, 1787), (153, 244, 1790), (151, 245, 1794), (150, 246, 1796), (148, 247, 1800), (146, 248, 1803), (144, 249, 1806), (143, 250, 1809), (141, 251, 1812), (139, 252, 1816), (137, 253, 1819), (136, 254, 1821), (134, 255, 1825), (132, 256, 1828), (130, 257, 1831), (128, 258, 1835), (127, 259, 1837), (125, 260, 1840), (124, 261, 1843), (123, 262, 1845), (122, 263, 1847), (120, 264, 1850), (119, 265, 1853), (118, 266, 1855), (117, 267, 1857), (116, 268, 1859), (115, 269, 1862), (114, 270, 1864), (113, 271, 1866), (112, 272, 1868), (111, 273, 1870), (110, 274, 1873), (109, 275, 1875), (108, 276, 1877), (108, 277, 1878), (107, 278, 1880), (106, 279, 1882), (106, 280, 1883), (105, 281, 1885), (105, 282, 1886), (104, 283, 1889), (104, 284, 1890), (104, 285, 1891), (103, 286, 1893), (103, 287, 1894), (102, 288, 1896), (102, 289, 1896), (101, 290, 1898), (101, 291, 1899), (100, 292, 1901), (100, 293, 1902), (99, 294, 1904), (99, 295, 1905), (98, 296, 1907), (98, 297, 1908), (97, 298, 1910), (97, 299, 1911), (97, 300, 1911), (96, 301, 1913), (96, 302, 1914), (95, 303, 1916), (95, 304, 1917), (94, 305, 1918), (94, 306, 1919), (94, 307, 1920), (93, 308, 1922), (93, 309, 1923), (92, 310, 1924), (92, 311, 1925), (92, 312, 1926), (91, 313, 1928), (91, 314, 1928), (90, 315, 1930), (90, 316, 1931), (90, 317, 1932), (89, 318, 1933), (89, 319, 1934), (88, 320, 1936), (88, 321, 1936), (88, 322, 1937), (87, 323, 1939), (87, 324, 1939), (86, 325, 1941), (86, 326, 1942), (86, 327, 1942), (85, 328, 1944), (85, 329, 1944), (85, 330, 1945), (84, 331, 1947), (84, 332, 1947), (83, 333, 1949), (83, 334, 1950), (83, 335, 1950), (82, 336, 1952), (82, 337, 1952), (82, 338, 1953), (81, 339, 1954), (81, 340, 1955), (81, 341, 1956), (80, 342, 1957), (80, 343, 1958), (80, 344, 1958), (79, 345, 1960), (79, 346, 1960), (79, 347, 1961), (78, 348, 1962), (78, 349, 1963), (78, 350, 1964), (77, 351, 1965), (77, 352, 1966), (77, 353, 1967), (77, 354, 1968), (76, 355, 1969), (76, 356, 1970), (76, 357, 1971), (76, 358, 1972), (75, 359, 1974), (75, 360, 1974), (75, 361, 1975), (74, 362, 1977), (74, 363, 1978), (74, 364, 1979), (74, 365, 1980), (73, 366, 1982), (73, 367, 1983), (73, 368, 1984), (72, 369, 1986), (72, 370, 1987), (72, 371, 1988), (72, 372, 1990), (71, 373, 1992), (71, 374, 1993), (71, 375, 1994), (70, 376, 1997), (70, 377, 1998), (70, 378, 1999), (69, 379, 2002), (69, 380, 2003), (69, 381, 2005), (68, 382, 2007), (68, 383, 2009), (68, 384, 2010), (67, 385, 2013), (67, 386, 2014), (67, 387, 2015), (66, 388, 2017), (66, 389, 2019), (66, 390, 2020), (65, 391, 2022), (65, 392, 2023), (65, 393, 2024), (64, 394, 2026), (64, 395, 2027), (64, 396, 2028), (63, 397, 2030), (63, 398, 2031), (63, 399, 2032), (62, 400, 2034), (62, 401, 2035), (61, 402, 2037), (61, 403, 2038), (61, 404, 2038), (60, 405, 2040), (60, 406, 2041), (60, 407, 2042), (59, 408, 2043), (59, 409, 2044), (58, 410, 2046), (58, 411, 2047), (57, 412, 2048), (57, 413, 2049), (57, 414, 2050), (56, 415, 2051), (56, 416, 2052), (55, 417, 2054), (55, 418, 2054), (54, 419, 2056), (54, 420, 2056), (54, 421, 2057), (53, 422, 2058), (53, 423, 2058), (53, 424, 2059), (53, 425, 2059), (52, 426, 2060), (52, 427, 2061), (52, 428, 2061), (52, 429, 2061), (51, 430, 2063), (51, 431, 2063), (51, 432, 2063), (51, 433, 2064), (50, 434, 2065), (50, 435, 2065), (50, 436, 2066), (49, 437, 2067), (49, 438, 2067), (49, 439, 2068), (49, 440, 2068), (48, 441, 2070), (48, 442, 2070), (48, 443, 2070), (47, 444, 2072), (47, 445, 2072), (47, 446, 2072), (47, 447, 2073), (46, 448, 2074), (46, 449, 2074), (46, 450, 2075), (45, 451, 2076), (45, 452, 2077), (45, 453, 2077), (44, 454, 2078), (44, 455, 2079), (44, 456, 2079), (44, 457, 2079), (43, 458, 2081), (43, 459, 2081), (43, 460, 2082), (42, 461, 2083), (42, 462, 2083), (42, 463, 2084), (41, 464, 2085), (41, 465, 2086), (41, 466, 2086), (40, 467, 2087), (40, 468, 2088), (40, 469, 2088), (39, 470, 2090), (39, 471, 2090), (39, 472, 2090), (39, 473, 2091), (38, 474, 2092), (38, 475, 2093), (38, 476, 2093), (37, 477, 2095), (37, 478, 2095), (37, 479, 2095), (37, 480, 2096), (36, 481, 2097), (36, 482, 2098), (36, 483, 2098), (36, 484, 2099), (35, 485, 2100), (35, 486, 2100), (35, 487, 2101), (34, 488, 2102), (34, 489, 2103), (34, 490, 2103), (34, 491, 2104), (33, 492, 2105), (33, 493, 2106), (33, 494, 2106), (33, 495, 2107), (33, 496, 2107), (33, 497, 2108), (32, 498, 2109), (32, 499, 2110), (32, 500, 2110), (32, 501, 2111), (32, 502, 2112), (31, 503, 2113), (31, 504, 2114), (31, 505, 2114), (31, 506, 2115), (31, 507, 2115), (31, 508, 2116), (30, 509, 2118), (30, 510, 2118), (30, 511, 2119), (30, 512, 2120), (30, 513, 2120), (30, 514, 2121), (29, 515, 2123), (29, 516, 2123), (29, 517, 2124), (29, 518, 2125), (29, 519, 2126), (29, 520, 2126), (28, 521, 2128), (28, 522, 2129), (28, 523, 2130), (28, 524, 2131), (28, 525, 2131), (28, 526, 2132), (28, 527, 2133), (27, 528, 2135), (27, 529, 2136), (27, 530, 2136), (27, 531, 2137), (27, 532, 2138), (27, 533, 2139), (26, 534, 2140), (26, 535, 2141), (26, 536, 2142), (26, 537, 2142), (26, 538, 2143), (26, 539, 2144), (26, 540, 2144), (25, 541, 2146), (25, 542, 2146), (25, 543, 2147), (25, 544, 2148), (25, 545, 2148), (25, 546, 2149), (25, 547, 2149), (24, 548, 2151), (24, 549, 2151), (24, 550, 2152), (24, 551, 2152), (24, 552, 2153), (24, 553, 2153), (24, 554, 2154), (23, 555, 2155), (23, 556, 2156), (23, 557, 2156), (23, 558, 2157), (23, 559, 2157), (23, 560, 2158), (23, 561, 2158), (23, 562, 2159), (22, 563, 2160), (22, 564, 2161), (22, 565, 2161), (22, 566, 2161), (22, 567, 2161), (22, 568, 2161), (22, 569, 2161), (22, 570, 2162), (22, 571, 2162), (22, 572, 2162), (22, 573, 2162), (22, 574, 2162), (22, 575, 2162), (21, 576, 2164), (21, 577, 2164), (21, 578, 2164), (21, 579, 2164), (21, 580, 2164), (21, 581, 2164), (21, 582, 2165), (21, 583, 2165), (21, 584, 2165), (21, 585, 2165), (21, 586, 2165), (21, 587, 2165), (21, 588, 2166), (21, 589, 2166), (20, 590, 2167), (20, 591, 2167), (20, 592, 2167), (20, 593, 2167), (20, 594, 2167), (20, 595, 2168), (20, 596, 2168), (20, 597, 2168), (20, 598, 2168), (20, 599, 2168), (20, 600, 2168), (20, 601, 2169), (20, 602, 2169), (20, 603, 2169), (20, 604, 2169), (19, 605, 2170), (19, 606, 2170), (19, 607, 2170), (19, 608, 2171), (19, 609, 2171), (19, 610, 2171), (19, 611, 2171), (19, 612, 2171), (19, 613, 2171), (19, 614, 2171), (19, 615, 2172), (19, 616, 2172), (19, 617, 2172), (19, 618, 2172), (19, 619, 2172), (19, 620, 2172), (18, 621, 2173), (18, 622, 2173), (18, 623, 2174), (18, 624, 2174), (18, 625, 2174), (18, 626, 2174), (18, 627, 2174), (18, 628, 2174), (18, 629, 2174), (18, 630, 2174), (18, 631, 2175), (18, 632, 2175), (18, 633, 2175), (18, 634, 2175), (18, 635, 2175), (18, 636, 2175), (18, 637, 2175), (18, 638, 2175), (18, 639, 2175), (18, 640, 2175), (18, 641, 2175), (18, 642, 2175), (18, 643, 2175), (18, 644, 2175), (19, 645, 2174), (19, 646, 2174), (19, 647, 2174), (19, 648, 2174), (19, 649, 2174), (19, 650, 2174), (19, 651, 2174), (19, 652, 2174), (19, 653, 2174), (19, 654, 2174), (20, 655, 2173), (20, 656, 2173), (20, 657, 2173), (20, 658, 2173), (20, 659, 2173), (20, 660, 2173), (20, 661, 2173), (20, 662, 2173), (20, 663, 2173), (20, 664, 2173), (21, 665, 2172), (21, 666, 2172), (21, 667, 2172), (21, 668, 2172), (21, 669, 2171), (21, 670, 2171), (21, 671, 2171), (21, 672, 2171), (21, 673, 2171), (22, 674, 2170), (22, 675, 2170), (22, 676, 2170), (22, 677, 2170), (22, 678, 2170), (22, 679, 2170), (22, 680, 2170), (22, 681, 2170), (22, 682, 2170), (23, 683, 2169), (23, 684, 2169), (23, 685, 2169), (23, 686, 2169), (23, 687, 2169), (23, 688, 2169), (23, 689, 2169), (23, 690, 2169), (23, 691, 2169), (24, 692, 2168), (24, 693, 2168), (24, 694, 2168), (24, 695, 2168), (24, 696, 2168), (24, 697, 2168), (24, 698, 2168), (24, 699, 2168), (24, 700, 2168), (25, 701, 2167), (25, 702, 2167), (25, 703, 2167), (25, 704, 2167), (25, 705, 2167), (25, 706, 2167), (25, 707, 2167), (25, 708, 2167), (25, 709, 2167), (25, 710, 2167), (25, 711, 2167), (25, 712, 2167), (25, 713, 2167), (25, 714, 2167), (25, 715, 2167), (25, 716, 2167), (25, 717, 2167), (25, 718, 2167), (25, 719, 2167), (25, 720, 2167), (25, 721, 2167), (25, 722, 2167), (25, 723, 2167), (25, 724, 2167), (25, 725, 2167), (25, 726, 2167), (25, 727, 2167), (25, 728, 2167), (25, 729, 2167), (25, 730, 2167), (25, 731, 2167), (25, 732, 2167), (25, 733, 2167), (25, 734, 2167), (25, 735, 2167), (25, 736, 2167), (25, 737, 2167), (25, 738, 2167), (25, 739, 2167), (25, 740, 2168), (25, 741, 2168), (25, 742, 2168), (25, 743, 2168), (25, 744, 2168), (25, 745, 2168), (25, 746, 2168), (25, 747, 2168), (25, 748, 2168), (25, 749, 2168), (25, 750, 2168), (25, 751, 2168), (25, 752, 2168), (25, 753, 2168), (25, 754, 2168), (25, 755, 2168), (25, 756, 2168), (25, 757, 2168), (24, 758, 2169), (24, 759, 2169), (24, 760, 2169), (24, 761, 2169), (24, 762, 2169), (24, 763, 2169), (24, 764, 2169), (24, 765, 2169), (24, 766, 2169), (24, 767, 2169), (24, 768, 2169), (24, 769, 2169), (24, 770, 2169), (24, 771, 2169), (24, 772, 2169), (24, 773, 2169), (24, 774, 2169), (24, 775, 2169), (24, 776, 2169), (24, 777, 2169), (24, 778, 2169), (24, 779, 2169), (24, 780, 2169), (24, 781, 2169), (24, 782, 2169), (24, 783, 2169), (25, 784, 2168), (25, 785, 2168), (25, 786, 2167), (25, 787, 2167), (25, 788, 2167), (25, 789, 2167), (25, 790, 2167), (25, 791, 2167), (25, 792, 2167), (25, 793, 2166), (25, 794, 2166), (25, 795, 2166), (25, 796, 2166), (25, 797, 2166), (25, 798, 2166), (25, 799, 2166), (26, 800, 2165), (26, 801, 2164), (26, 802, 2164), (26, 803, 2164), (26, 804, 2164), (26, 805, 2164), (26, 806, 2164), (26, 807, 2164), (26, 808, 2163), (26, 809, 2163), (26, 810, 2163), (26, 811, 2163), (26, 812, 2163), (26, 813, 2163), (26, 814, 2163), (27, 815, 2161), (27, 816, 2161), (27, 817, 2161), (27, 818, 2161), (27, 819, 2161), (27, 820, 2161), (27, 821, 2160), (27, 822, 2160), (27, 823, 2160), (27, 824, 2160), (27, 825, 2160), (27, 826, 2160), (27, 827, 2160), (27, 828, 2159), (28, 829, 2158), (28, 830, 2158), (28, 831, 2158), (28, 832, 2158), (28, 833, 2158), (28, 834, 2157), (28, 835, 2157), (28, 836, 2157), (28, 837, 2157), (28, 838, 2157), (28, 839, 2157), (28, 840, 2156), (28, 841, 2156), (28, 842, 2156), (28, 843, 2156), (29, 844, 2155), (29, 845, 2155), (29, 846, 2154), (29, 847, 2154), (29, 848, 2154), (29, 849, 2154), (29, 850, 2154), (29, 851, 2153), (29, 852, 2153), (29, 853, 2153), (29, 854, 2153), (29, 855, 2153), (29, 856, 2153), (29, 857, 2152), (29, 858, 2152), (29, 859, 2152), (29, 860, 2152), (29, 861, 2152), (29, 862, 2151), (29, 863, 2151), (29, 864, 2151), (28, 865, 2152), (28, 866, 2152), (28, 867, 2151), (28, 868, 2151), (28, 869, 2151), (28, 870, 2151), (28, 871, 2151), (28, 872, 2150), (28, 873, 2150), (28, 874, 2150), (28, 875, 2150), (28, 876, 2150), (28, 877, 2149), (28, 878, 2149), (28, 879, 2149), (28, 880, 2149), (28, 881, 2148), (28, 882, 2148), (28, 883, 2148), (28, 884, 2148), (28, 885, 2148), (28, 886, 2147), (28, 887, 2147), (28, 888, 2147), (28, 889, 2147), (28, 890, 2146), (28, 891, 2146), (28, 892, 2146), (28, 893, 2146), (28, 894, 2146), (27, 895, 2146), (27, 896, 2146), (27, 897, 2146), (27, 898, 2146), (27, 899, 2145), (27, 900, 2145), (27, 901, 2145), (27, 902, 2145), (27, 903, 2144), (27, 904, 2144), (27, 905, 2144), (27, 906, 2144), (27, 907, 2143), (27, 908, 2143), (27, 909, 2143), (27, 910, 2142), (27, 911, 2142), (27, 912, 2142), (27, 913, 2142), (27, 914, 2141), (27, 915, 2141), (27, 916, 2141), (27, 917, 2141), (27, 918, 2140), (27, 919, 2140), (27, 920, 2140), (27, 921, 2139), (27, 922, 2139), (27, 923, 2138), (27, 924, 2137), (27, 925, 2137), (27, 926, 2136), (27, 927, 2135), (27, 928, 2134), (27, 929, 2133), (27, 930, 2132), (27, 931, 2131), (27, 932, 2131), (27, 933, 2130), (27, 934, 2129), (27, 935, 2128), (27, 936, 2127), (28, 937, 2125), (28, 938, 2125), (28, 939, 2124), (28, 940, 2123), (28, 941, 2122), (28, 942, 2122), (28, 943, 2121), (28, 944, 2120), (28, 945, 2119), (28, 946, 2119), (28, 947, 2118), (28, 948, 2117), (28, 949, 2117), (28, 950, 2116), (28, 951, 2115), (28, 952, 2115), (28, 953, 2114), (29, 954, 2113), (29, 955, 2112), (29, 956, 2111), (29, 957, 2111), (29, 958, 2110), (29, 959, 2110), (29, 960, 2109), (29, 961, 2109), (29, 962, 2108), (29, 963, 2108), (29, 964, 2107), (29, 965, 2107), (29, 966, 2106), (29, 967, 2106), (29, 968, 2105), (29, 969, 2105), (30, 970, 2103), (30, 971, 2103), (30, 972, 2102), (30, 973, 2102), (30, 974, 2101), (30, 975, 2101), (30, 976, 2100), (30, 977, 2100), (30, 978, 2100), (30, 979, 2099), (30, 980, 2099), (30, 981, 2098), (30, 982, 2098), (30, 983, 2098), (30, 984, 2097), (30, 985, 2097), (31, 986, 2095), (31, 987, 2095), (31, 988, 2095), (31, 989, 2094), (31, 990, 2094), (31, 991, 2094), (31, 992, 2093), (31, 993, 2093), (31, 994, 2093), (31, 995, 2092), (31, 996, 2092), (31, 997, 2092), (31, 998, 2092), (31, 999, 2091), (31, 1000, 2091), (31, 1001, 2091), (31, 1002, 2091), (31, 1003, 2091), (31, 1004, 2090), (31, 1005, 2090), (30, 1006, 2091), (30, 1007, 2091), (30, 1008, 2090), (30, 1009, 2090), (30, 1010, 2090), (30, 1011, 2090), (30, 1012, 2089), (30, 1013, 2089), (30, 1014, 2089), (30, 1015, 2089), (30, 1016, 2088), (30, 1017, 2088), (30, 1018, 2088), (30, 1019, 2088), (30, 1020, 2087), (30, 1021, 2087), (30, 1022, 2087), (30, 1023, 2087), (30, 1024, 2086), (30, 1025, 2086), (30, 1026, 2086), (30, 1027, 2086), (30, 1028, 2085), (30, 1029, 2085), (30, 1030, 2085), (29, 1031, 2086), (29, 1032, 2085), (29, 1033, 2085), (29, 1034, 2085), (29, 1035, 2085), (29, 1036, 2084), (29, 1037, 2084), (29, 1038, 2084), (29, 1039, 2084), (29, 1040, 2084), (29, 1041, 2083), (29, 1042, 2083), (29, 1043, 2083), (29, 1044, 2083), (29, 1045, 2082), (29, 1046, 2082), (29, 1047, 2082), (29, 1048, 2081), (29, 1049, 2081), (29, 1050, 2081), (29, 1051, 2081), (29, 1052, 2080), (29, 1053, 2080), (29, 1054, 2080), (29, 1055, 2080), (28, 1056, 2080), (28, 1057, 2080), (28, 1058, 2080), (28, 1059, 2080), (28, 1060, 2079), (28, 1061, 2079), (28, 1062, 2079), (28, 1063, 2079), (28, 1064, 2078), (28, 1065, 2078), (28, 1066, 2077), (28, 1067, 2077), (28, 1068, 2076), (28, 1069, 2075), (28, 1070, 2075), (28, 1071, 2074), (28, 1072, 2073), (29, 1073, 2071), (29, 1074, 2071), (29, 1075, 2070), (29, 1076, 2069), (29, 1077, 2068), (29, 1078, 2067), (29, 1079, 2066), (29, 1080, 2065), (29, 1081, 2064), (29, 1082, 2063), (29, 1083, 2062), (29, 1084, 2061), (29, 1085, 2060), (29, 1086, 2059), (29, 1087, 2058), (29, 1088, 2057), (29, 1089, 2055), (29, 1090, 2054), (29, 1091, 2053), (29, 1092, 2051), (29, 1093, 2050), (29, 1094, 2049), (30, 1095, 2046), (30, 1096, 2045), (30, 1097, 2043), (30, 1098, 2041), (30, 1099, 2040), (30, 1100, 2038), (30, 1101, 2037), (30, 1102, 2035), (30, 1103, 2034), (30, 1104, 2033), (30, 1105, 2031), (30, 1106, 2030), (30, 1107, 2029), (30, 1108, 2028), (30, 1109, 2026), (30, 1110, 2025), (30, 1111, 2024), (30, 1112, 2023), (30, 1113, 2022), (30, 1114, 2021), (30, 1115, 2020), (31, 1116, 2018), (31, 1117, 2017), (31, 1118, 2016), (31, 1119, 2015), (31, 1120, 2015), (31, 1121, 2014), (31, 1122, 2013), (31, 1123, 2012), (31, 1124, 2011), (31, 1125, 2010), (31, 1126, 2010), (31, 1127, 2009), (31, 1128, 2008), (31, 1129, 2008), (31, 1130, 2007), (31, 1131, 2006), (31, 1132, 2006), (31, 1133, 2005), (31, 1134, 2004), (31, 1135, 2004), (31, 1136, 2003), (31, 1137, 2003), (31, 1138, 2002), (31, 1139, 2002), (31, 1140, 2001), (31, 1141, 2001), (31, 1142, 2000), (31, 1143, 2000), (31, 1144, 1999), (31, 1145, 1999), (31, 1146, 1998), (31, 1147, 1998), (31, 1148, 1998), (31, 1149, 1997), (31, 1150, 1997), (31, 1151, 1996), (31, 1152, 1996), (31, 1153, 1995), (31, 1154, 1995), (31, 1155, 1994), (31, 1156, 1993), (31, 1157, 1993), (31, 1158, 1992), (31, 1159, 1992), (31, 1160, 1991), (31, 1161, 1991), (31, 1162, 1990), (31, 1163, 1990), (31, 1164, 1989), (31, 1165, 1988), (31, 1166, 1988), (31, 1167, 1987), (31, 1168, 1987), (31, 1169, 1986), (31, 1170, 1985), (31, 1171, 1985), (31, 1172, 1984), (31, 1173, 1983), (31, 1174, 1983), (31, 1175, 1982), (31, 1176, 1981), (31, 1177, 1981), (31, 1178, 1980), (31, 1179, 1979), (31, 1180, 1978), (31, 1181, 1978), (30, 1182, 1978), (30, 1183, 1977), (30, 1184, 1976), (30, 1185, 1976), (30, 1186, 1975), (30, 1187, 1974), (30, 1188, 1973), (30, 1189, 1972), (30, 1190, 1971), (30, 1191, 1971), (30, 1192, 1970), (30, 1193, 1969), (30, 1194, 1968), (30, 1195, 1967), (30, 1196, 1966), (30, 1197, 1965), (30, 1198, 1964), (30, 1199, 1963), (30, 1200, 1962), (30, 1201, 1961), (30, 1202, 1960), (30, 1203, 1959), (30, 1204, 1958), (30, 1205, 1957), (30, 1206, 1956), (30, 1207, 1955), (30, 1208, 1954), (30, 1209, 1953), (30, 1210, 1953), (30, 1211, 1952), (30, 1212, 1951), (30, 1213, 1950), (30, 1214, 1949), (30, 1215, 1949), (30, 1216, 1948), (30, 1217, 1947), (30, 1218, 1947), (30, 1219, 1946), (30, 1220, 1945), (30, 1221, 1945), (31, 1222, 1943), (31, 1223, 1942), (31, 1224, 1942), (31, 1225, 1941), (31, 1226, 1940), (31, 1227, 1940), (31, 1228, 1939), (31, 1229, 1939), (31, 1230, 1938), (31, 1231, 1937), (31, 1232, 1937), (31, 1233, 1936), (31, 1234, 1936), (31, 1235, 1935), (31, 1236, 1935), (31, 1237, 1934), (31, 1238, 1934), (31, 1239, 1933), (31, 1240, 1933), (31, 1241, 1932), (31, 1242, 1932), (31, 1243, 1931), (31, 1244, 1931), (31, 1245, 1930), (31, 1246, 1930), (32, 1247, 1928), (32, 1248, 1928), (32, 1249, 1927), (32, 1250, 1927), (32, 1251, 1927), (32, 1252, 1926), (32, 1253, 1926), (32, 1254, 1925), (32, 1255, 1925), (32, 1256, 1924), (32, 1257, 1924), (32, 1258, 1924), (32, 1259, 1923), (32, 1260, 1923), (32, 1261, 1922), (32, 1262, 1922), (32, 1263, 1922), (32, 1264, 1921), (32, 1265, 1921), (32, 1266, 1921), (32, 1267, 1920), (32, 1268, 1920), (32, 1269, 1920), (33, 1270, 1918), (33, 1271, 1918), (33, 1272, 1918), (33, 1273, 1917), (33, 1274, 1917), (33, 1275, 1917), (33, 1276, 1916), (33, 1277, 1916), (33, 1278, 1916), (33, 1279, 1915), (33, 1280, 1915), (33, 1281, 1915), (33, 1282, 1915), (33, 1283, 1915), (33, 1284, 1914), (33, 1285, 1914), (33, 1286, 1914), (33, 1287, 1914), (33, 1288, 1914), (33, 1289, 1914), (33, 1290, 1913), (33, 1291, 1913), (33, 1292, 1913), (34, 1293, 1912), (34, 1294, 1912), (34, 1295, 1912), (34, 1296, 1911), (34, 1297, 1911), (34, 1298, 1911), (34, 1299, 1911), (34, 1300, 1911), (34, 1301, 1911), (34, 1302, 1910), (34, 1303, 1910), (34, 1304, 1910), (34, 1305, 1910), (34, 1306, 1910), (34, 1307, 1910), (34, 1308, 1909), (34, 1309, 1909), (34, 1310, 1909), (34, 1311, 1909), (34, 1312, 1909), (34, 1313, 1908), (35, 1314, 1907), (35, 1315, 1907), (35, 1316, 1907), (35, 1317, 1907), (35, 1318, 1907), (35, 1319, 1906), (35, 1320, 1906), (35, 1321, 1906), (35, 1322, 1906), (35, 1323, 1906), (35, 1324, 1905), (35, 1325, 1905), (35, 1326, 1905), (35, 1327, 1905), (35, 1328, 1905), (35, 1329, 1904), (35, 1330, 1904), (35, 1331, 1904), (35, 1332, 1904), (35, 1333, 1904), (36, 1334, 1902), (36, 1335, 1902), (36, 1336, 1902), (36, 1337, 1902), (36, 1338, 1902), (36, 1339, 1902), (36, 1340, 1901), (36, 1341, 1901), (36, 1342, 1901), (36, 1343, 1901), (36, 1344, 1901), (36, 1345, 1900), (36, 1346, 1900), (36, 1347, 1900), (36, 1348, 1900), (36, 1349, 1899), (36, 1350, 1899), (36, 1351, 1899), (36, 1352, 1899), (36, 1353, 1898), (36, 1354, 1898), (37, 1355, 1897), (37, 1356, 1897), (37, 1357, 1896), (37, 1358, 1896), (37, 1359, 1896), (37, 1360, 1895), (37, 1361, 1895), (37, 1362, 1895), (37, 1363, 1894), (37, 1364, 1894), (37, 1365, 1894), (37, 1366, 1893), (37, 1367, 1893), (37, 1368, 1893), (37, 1369, 1892), (37, 1370, 1892), (37, 1371, 1892), (37, 1372, 1891), (37, 1373, 1891), (37, 1374, 1891), (37, 1375, 1890), (37, 1376, 1890), (37, 1377, 1890), (37, 1378, 1889), (37, 1379, 1889), (38, 1380, 1887), (38, 1381, 1887), (38, 1382, 1887), (38, 1383, 1886), (38, 1384, 1886), (38, 1385, 1885), (38, 1386, 1885), (38, 1387, 1885), (38, 1388, 1884), (38, 1389, 1884), (38, 1390, 1883), (38, 1391, 1883), (38, 1392, 1882), (38, 1393, 1882), (38, 1394, 1881), (38, 1395, 1881), (38, 1396, 1880), (38, 1397, 1880), (38, 1398, 1879), (38, 1399, 1879), (38, 1400, 1878), (38, 1401, 1878), (38, 1402, 1877), (38, 1403, 1877), (39, 1404, 1875), (39, 1405, 1875), (39, 1406, 1874), (39, 1407, 1874), (39, 1408, 1873), (39, 1409, 1873), (39, 1410, 1872), (39, 1411, 1872), (39, 1412, 1871), (39, 1413, 1870), (39, 1414, 1870), (39, 1415, 1869), (39, 1416, 1868), (39, 1417, 1868), (39, 1418, 1867), (39, 1419, 1867), (39, 1420, 1866), (39, 1421, 1865), (39, 1422, 1865), (39, 1423, 1864), (39, 1424, 1863), (40, 1425, 1862), (40, 1426, 1861), (40, 1427, 1861), (40, 1428, 1860), (40, 1429, 1860), (40, 1430, 1859), (40, 1431, 1858), (40, 1432, 1858), (40, 1433, 1857), (40, 1434, 1857), (40, 1435, 1856), (41, 1436, 1855), (41, 1437, 1854), (41, 1438, 1854), (41, 1439, 1853), (41, 1440, 1853), (41, 1441, 1852), (41, 1442, 1852), (41, 1443, 1852), (41, 1444, 1851), (41, 1445, 1851), (42, 1446, 1849), (42, 1447, 1849), (42, 1448, 1848), (42, 1449, 1848), (42, 1450, 1847), (42, 1451, 1847), (42, 1452, 1847), (42, 1453, 1846), (42, 1454, 1846), (43, 1455, 1844), (43, 1456, 1844), (43, 1457, 1843), (43, 1458, 1843), (43, 1459, 1843), (43, 1460, 1842), (43, 1461, 1842), (43, 1462, 1842), (43, 1463, 1841), (44, 1464, 1840), (44, 1465, 1839), (44, 1466, 1839), (44, 1467, 1839), (44, 1468, 1838), (44, 1469, 1838), (44, 1470, 1838), (44, 1471, 1837), (44, 1472, 1837), (44, 1473, 1836), (45, 1474, 1835), (45, 1475, 1835), (45, 1476, 1834), (45, 1477, 1834), (45, 1478, 1834), (45, 1479, 1833), (45, 1480, 1833), (45, 1481, 1833), (45, 1482, 1833), (46, 1483, 1831), (46, 1484, 1831), (46, 1485, 1831), (46, 1486, 1830), (46, 1487, 1830), (46, 1488, 1830), (46, 1489, 1829), (46, 1490, 1829), (46, 1491, 1829), (46, 1492, 1828), (47, 1493, 1827), (47, 1494, 1827), (47, 1495, 1827), (47, 1496, 1827), (47, 1497, 1826), (46, 1498, 1827), (46, 1499, 1827), (46, 1500, 1827), (46, 1501, 1827), (46, 1502, 1827), (46, 1503, 1827), (46, 1504, 1826), (46, 1505, 1826), (46, 1506, 1826), (46, 1507, 1826), (46, 1508, 1826), (46, 1509, 1826), (46, 1510, 1826), (46, 1511, 1825), (46, 1512, 1825), (46, 1513, 1825), (46, 1514, 1825), (46, 1515, 1825), (46, 1516, 1825), (45, 1517, 1826), (45, 1518, 1825), (45, 1519, 1825), (45, 1520, 1825), (45, 1521, 1825), (45, 1522, 1825), (45, 1523, 1825), (45, 1524, 1825), (45, 1525, 1824), (45, 1526, 1824), (45, 1527, 1824), (45, 1528, 1824), (45, 1529, 1824), (45, 1530, 1824), (45, 1531, 1824), (45, 1532, 1823), (45, 1533, 1823), (45, 1534, 1823), (44, 1535, 1824), (44, 1536, 1824), (44, 1537, 1824), (44, 1538, 1824), (44, 1539, 1823), (44, 1540, 1823), (44, 1541, 1823), (44, 1542, 1823), (44, 1543, 1823), (44, 1544, 1823), (44, 1545, 1823), (44, 1546, 1822), (44, 1547, 1822), (44, 1548, 1822), (44, 1549, 1822), (44, 1550, 1822), (44, 1551, 1822), (43, 1552, 1823), (43, 1553, 1823), (43, 1554, 1822), (43, 1555, 1822), (43, 1556, 1822), (43, 1557, 1822), (43, 1558, 1822), (43, 1559, 1822), (43, 1560, 1822), (43, 1561, 1821), (43, 1562, 1821), (43, 1563, 1821), (43, 1564, 1821), (43, 1565, 1821), (43, 1566, 1821), (43, 1567, 1821), (43, 1568, 1821), (43, 1569, 1820), (43, 1570, 1820), (43, 1571, 1820), (43, 1572, 1820), (43, 1573, 1820), (43, 1574, 1820), (43, 1575, 1820), (43, 1576, 1820), (43, 1577, 1820), (43, 1578, 1819), (43, 1579, 1819), (43, 1580, 1819), (43, 1581, 1819), (43, 1582, 1819), (43, 1583, 1819), (44, 1584, 1818), (44, 1585, 1818), (44, 1586, 1818), (44, 1587, 1818), (44, 1588, 1817), (44, 1589, 1817), (44, 1590, 1817), (44, 1591, 1817), (44, 1592, 1817), (44, 1593, 1817), (44, 1594, 1817), (44, 1595, 1817), (44, 1596, 1817), (44, 1597, 1816), (44, 1598, 1816), (44, 1599, 1816), (44, 1600, 1816), (44, 1601, 1816), (44, 1602, 1816), (44, 1603, 1816), (44, 1604, 1816), (44, 1605, 1816), (44, 1606, 1815), (45, 1607, 1814), (45, 1608, 1814), (45, 1609, 1814), (45, 1610, 1814), (45, 1611, 1814), (45, 1612, 1814), (45, 1613, 1814), (45, 1614, 1814), (45, 1615, 1813), (45, 1616, 1813), (45, 1617, 1813), (45, 1618, 1813), (45, 1619, 1813), (45, 1620, 1813), (45, 1621, 1813), (45, 1622, 1813), (45, 1623, 1813), (45, 1624, 1812), (45, 1625, 1812), (45, 1626, 1812), (45, 1627, 1812), (45, 1628, 1812), (45, 1629, 1812), (46, 1630, 1811), (46, 1631, 1811), (46, 1632, 1811), (46, 1633, 1810), (46, 1634, 1810), (46, 1635, 1810), (46, 1636, 1810), (46, 1637, 1810), (46, 1638, 1810), (46, 1639, 1810), (46, 1640, 1809), (46, 1641, 1809), (46, 1642, 1809), (46, 1643, 1809), (46, 1644, 1808), (45, 1645, 1809), (45, 1646, 1809), (45, 1647, 1809), (45, 1648, 1809), (45, 1649, 1808), (45, 1650, 1808), (45, 1651, 1808), (45, 1652, 1808), (45, 1653, 1807), (45, 1654, 1807), (45, 1655, 1807), (45, 1656, 1807), (45, 1657, 1807), (45, 1658, 1806), (45, 1659, 1806), (45, 1660, 1806), (45, 1661, 1806), (45, 1662, 1805), (45, 1663, 1805), (45, 1664, 1805), (45, 1665, 1805), (45, 1666, 1804), (44, 1667, 1805), (44, 1668, 1805), (44, 1669, 1805), (44, 1670, 1804), (44, 1671, 1804), (44, 1672, 1804), (44, 1673, 1804), (44, 1674, 1803), (44, 1675, 1803), (44, 1676, 1803), (44, 1677, 1803), (44, 1678, 1802), (44, 1679, 1802), (44, 1680, 1802), (44, 1681, 1801), (44, 1682, 1801), (44, 1683, 1801), (44, 1684, 1801), (44, 1685, 1800), (44, 1686, 1800), (44, 1687, 1800), (43, 1688, 1801), (43, 1689, 1800), (43, 1690, 1800), (43, 1691, 1800), (43, 1692, 1799), (43, 1693, 1799), (43, 1694, 1799), (43, 1695, 1799), (43, 1696, 1798), (43, 1697, 1798), (43, 1698, 1798), (43, 1699, 1797), (43, 1700, 1797), (43, 1701, 1797), (43, 1702, 1796), (43, 1703, 1796), (43, 1704, 1796), (43, 1705, 1795), (43, 1706, 1795), (43, 1707, 1795), (42, 1708, 1795), (42, 1709, 1795), (43, 1710, 1794), (43, 1711, 1793), (43, 1712, 1793), (44, 1713, 1792), (44, 1714, 1791), (44, 1715, 1791), (44, 1716, 1791), (45, 1717, 1789), (45, 1718, 1789), (45, 1719, 1788), (45, 1720, 1788), (46, 1721, 1787), (46, 1722, 1786), (46, 1723, 1786), (46, 1724, 1785), (46, 1725, 1785), (47, 1726, 1784), (47, 1727, 1783), (47, 1728, 1783), (47, 1729, 1782), (48, 1730, 1781), (48, 1731, 1780), (48, 1732, 1780), (48, 1733, 1779), (49, 1734, 1778), (49, 1735, 1778), (49, 1736, 1777), (49, 1737, 1777), (50, 1738, 1775), (50, 1739, 1775), (50, 1740, 1774), (50, 1741, 1774), (50, 1742, 1773), (51, 1743, 1772), (51, 1744, 1771), (51, 1745, 1771), (51, 1746, 1770), (52, 1747, 1769), (52, 1748, 1768), (52, 1749, 1768), (52, 1750, 1767), (52, 1751, 1767), (53, 1752, 1765), (53, 1753, 1765), (53, 1754, 1764), (53, 1755, 1764), (54, 1756, 1762), (54, 1757, 1762), (54, 1758, 1762), (54, 1759, 1761), (54, 1760, 1761), (55, 1761, 1759), (55, 1762, 1759), (55, 1763, 1758), (55, 1764, 1758), (55, 1765, 1758), (56, 1766, 1756), (56, 1767, 1756), (56, 1768, 1755), (56, 1769, 1755), (57, 1770, 1754), (57, 1771, 1753), (57, 1772, 1753), (57, 1773, 1752), (57, 1774, 1752), (58, 1775, 1751), (58, 1776, 1750), (58, 1777, 1750), (58, 1778, 1750), (58, 1779, 1749), (59, 1780, 1748), (59, 1781, 1748), (59, 1782, 1747), (59, 1783, 1747), (59, 1784, 1746), (59, 1785, 1746), (59, 1786, 1746), (59, 1787, 1745), (59, 1788, 1745), (59, 1789, 1745), (59, 1790, 1744), (59, 1791, 1744), (59, 1792, 1744), (59, 1793, 1743), (59, 1794, 1743), (59, 1795, 1743), (59, 1796, 1742), (59, 1797, 1742), (59, 1798, 1742), (59, 1799, 1742), (59, 1800, 1741), (59, 1801, 1741), (59, 1802, 1741), (59, 1803, 1740), (59, 1804, 1740), (59, 1805, 1740), (59, 1806, 1739), (59, 1807, 1739), (59, 1808, 1739), (59, 1809, 1739), (59, 1810, 1738), (59, 1811, 1738), (59, 1812, 1738), (60, 1813, 1736), (60, 1814, 1736), (60, 1815, 1736), (60, 1816, 1736), (60, 1817, 1735), (60, 1818, 1735), (60, 1819, 1735), (60, 1820, 1734), (60, 1821, 1734), (60, 1822, 1734), (60, 1823, 1734), (60, 1824, 1733), (60, 1825, 1733), (60, 1826, 1733), (60, 1827, 1733), (60, 1828, 1732), (60, 1829, 1732), (60, 1830, 1732), (60, 1831, 1732), (60, 1832, 1731), (60, 1833, 1731), (60, 1834, 1731), (60, 1835, 1731), (60, 1836, 1730), (60, 1837, 1730), (60, 1838, 1730), (60, 1839, 1730), (60, 1840, 1730), (60, 1841, 1729), (60, 1842, 1729), (60, 1843, 1729), (60, 1844, 1729), (61, 1845, 1727), (61, 1846, 1727), (61, 1847, 1727), (61, 1848, 1727), (61, 1849, 1726), (61, 1850, 1726), (61, 1851, 1726), (61, 1852, 1726), (61, 1853, 1726), (62, 1854, 1724), (62, 1855, 1724), (62, 1856, 1724), (63, 1857, 1723), (63, 1858, 1722), (64, 1859, 1721), (64, 1860, 1721), (65, 1861, 1720), (65, 1862, 1719), (65, 1863, 1719), (66, 1864, 1718), (66, 1865, 1718), (67, 1866, 1716), (67, 1867, 1716), (68, 1868, 1715), (68, 1869, 1714), (68, 1870, 1714), (69, 1871, 1713), (69, 1872, 1713), (70, 1873, 1711), (70, 1874, 1711), (70, 1875, 1711), (71, 1876, 1710), (71, 1877, 1709), (72, 1878, 1708), (72, 1879, 1708), (73, 1880, 1707), (73, 1881, 1706), (74, 1882, 1705), (74, 1883, 1705), (74, 1884, 1705), (75, 1885, 1703), (75, 1886, 1703), (76, 1887, 1702), (76, 1888, 1702), (77, 1889, 1700), (77, 1890, 1700), (77, 1891, 1700), (78, 1892, 1698), (78, 1893, 1698), (79, 1894, 1697), (79, 1895, 1697), (80, 1896, 1695), (80, 1897, 1695), (80, 1898, 1695), (81, 1899, 1694), (81, 1900, 1693), (82, 1901, 1692), (82, 1902, 1692), (83, 1903, 1690), (83, 1904, 1690), (84, 1905, 1689), (84, 1906, 1689), (84, 1907, 1688), (85, 1908, 1687), (85, 1909, 1687), (86, 1910, 1686), (86, 1911, 1685), (87, 1912, 1684), (87, 1913, 1684), (88, 1914, 1683), (88, 1915, 1682), (88, 1916, 1682), (89, 1917, 1681), (89, 1918, 1680), (90, 1919, 1679), (90, 1920, 1679), (91, 1921, 1678), (91, 1922, 1677), (92, 1923, 1676), (92, 1924, 1676), (93, 1925, 1674), (93, 1926, 1674), (94, 1927, 1672), (94, 1928, 1672), (95, 1929, 1671), (95, 1930, 1670), (96, 1931, 1669), (97, 1932, 1667), (97, 1933, 1667), (98, 1934, 1665), (99, 1935, 1664), (99, 1936, 1664), (100, 1937, 1662), (101, 1938, 1661), (101, 1939, 1660), (102, 1940, 1659), (103, 1941, 1657), (103, 1942, 1657), (104, 1943, 1655), (105, 1944, 1653), (106, 1945, 1652), (107, 1946, 1650), (107, 1947, 1650), (108, 1948, 1648), (109, 1949, 1647), (110, 1950, 1645), (111, 1951, 1643), (112, 1952, 1642), (112, 1953, 1641), (113, 1954, 1640), (114, 1955, 1638), (115, 1956, 1636), (116, 1957, 1635), (117, 1958, 1633), (118, 1959, 1631), (119, 1960, 1630), (120, 1961, 1628), (121, 1962, 1626), (122, 1963, 1624), (123, 1964, 1623), (125, 1965, 1620), (126, 1966, 1618), (127, 1967, 1616), (137, 1968, 1605), (148, 1969, 1593), (159, 1970, 1582), (170, 1971, 1570), (181, 1972, 1558), (192, 1973, 1546), (203, 1974, 1534), (213, 1975, 1523), (219, 1976, 1515), (224, 1977, 1508), (230, 1978, 1501), (235, 1979, 1494), (241, 1980, 1487), (246, 1981, 1480), (252, 1982, 1473), (258, 1983, 1465), (263, 1984, 1459), (269, 1985, 1451), (275, 1986, 1444), (281, 1987, 1436), (287, 1988, 1429), (292, 1989, 1422), (300, 1990, 1413), (311, 1991, 1401), (322, 1992, 1388), (333, 1993, 1376), (344, 1994, 1364), (355, 1995, 1351), (368, 1996, 1337), (381, 1997, 1322), (384, 1998, 1318), (387, 1999, 1313), (390, 2000, 1308), (393, 2001, 1303), (396, 2002, 1299), (399, 2003, 1294), (402, 2004, 1289), (405, 2005, 1284), (409, 2006, 1278), (412, 2007, 1273), (415, 2008, 1267), (419, 2009, 1261), (422, 2010, 1256), (426, 2011, 1249), (429, 2012, 1244), (433, 2013, 1237), (437, 2014, 1231), (440, 2015, 1225), (444, 2016, 1218), (448, 2017, 1211), (452, 2018, 1204), (456, 2019, 1197), (460, 2020, 1189), (465, 2021, 1178), (470, 2022, 1168), (477, 2023, 1155), (483, 2024, 1143), (490, 2025, 1130), (497, 2026, 1117), (504, 2027, 1104), (511, 2028, 1091), (518, 2029, 1078), (525, 2030, 1065), (532, 2031, 1052), (540, 2032, 1037), (548, 2033, 1023), (554, 2034, 1005), (559, 2035, 979), (565, 2036, 952), (570, 2037, 927), (576, 2038, 905), (581, 2039, 895), (586, 2040, 885), (591, 2041, 875), (596, 2042, 865), (601, 2043, 856), (606, 2044, 846), (611, 2045, 837), (616, 2046, 828), (621, 2047, 818), (626, 2048, 809), (630, 2049, 801), (635, 2050, 792), (638, 2051, 785), (641, 2052, 778), (644, 2053, 772), (647, 2054, 765), (650, 2055, 758), (653, 2056, 752), (655, 2057, 746), (658, 2058, 740), (660, 2059, 732), (663, 2060, 724), (665, 2061, 716), (668, 2062, 708), (670, 2063, 701), (672, 2064, 694), (675, 2065, 686), (677, 2066, 679), (679, 2067, 672), (682, 2068, 663), (685, 2069, 654), (688, 2070, 645), (691, 2071, 636), (694, 2072, 627), (697, 2073, 617), (701, 2074, 606), (704, 2075, 597), (708, 2076, 586), (712, 2077, 574), (715, 2078, 564), (719, 2079, 552), (729, 2080, 534), (740, 2081, 515), (751, 2082, 496), (763, 2083, 475), (775, 2084, 454), (787, 2085, 432), (800, 2086, 409), (811, 2087, 388), (822, 2088, 367), (833, 2089, 345), (844, 2090, 322), (856, 2091, 298), (868, 2092, 271), (881, 2093, 214), (909, 2094, 138), (960, 2095, 29)], ['881,2093,719,2079,625,2047,460,2020,380,1996,127,1967,95,1930,61,1853,59,1780,43,1707,46,1483,30,1221,29,864,25,701,18,644,33,492,83,333,125,260,252,184,457,126,546,111,705,108,1319,109,1482,115,1565,139,1838,178,1982,274,2060,372,2110,421,2134,486,2182,564,2192,631,2192,785,2173,894,2122,995,2106,1063,2056,1108,1967,1231,1947,1279,1933,1356,1872,1497,1855,1639,1830,1726,1793,1820,1767,1924,1735,1975,1648,2020,1481,2037,1397,2058,1138,2092'])], 'temp/1738690562_598972_917877156_a9c2d4b99270c9302def4ed40606e685.jpg']} nb pixel non reg : 3692295 nb pixel common : 3663520 proportion of common points : 0.992206744044016 [('test release memory', 'SUCCESS', True), ('test detect objet', 'SUCCESS', True), ('test polygone', 'SUCCESS', True)] res_total : True #&_# TEST SUCCEEDED #&_# : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_2df2c27324c97325c9ce9b0c963d4713663fd3d9 SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_2df2c27324c97325c9ce9b0c963d4713663fd3d9','{"mask_detection": "success"}','1','http://marlene.fotonower-preprod.com/job/2025/February/04022025/python_test3//data_2/data_log/job/2025/February/04022025/python_test3/log-python3----short_python3--v--marlene-18:35:01.txt','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.24826931953430176 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Tue Feb 4 18:36:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1738690589_598972_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1738690589_598972_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 (640, 960, 3) ERROR in datou_step_exec, will save and exit ! CUDA out of memory. Tried to allocate 768.00 MiB (GPU 0; 10.76 GiB total capacity; 1.20 GiB already allocated; 74.88 MiB free; 1.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2430, in datou_step_exec return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 396, in datou_step_sam masks = mask_generator.generate(image) File "/home/admin/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/automatic_mask_generator.py", line 163, in generate mask_data = self._generate_masks(image) File "/home/admin/workarea/install/segment-anything/segment_anything/automatic_mask_generator.py", line 206, in _generate_masks crop_data = self._process_crop(image, crop_box, layer_idx, orig_size) File "/home/admin/workarea/install/segment-anything/segment_anything/automatic_mask_generator.py", line 236, in _process_crop self.predictor.set_image(cropped_im) File "/home/admin/workarea/install/segment-anything/segment_anything/predictor.py", line 60, in set_image self.set_torch_image(input_image_torch, image.shape[:2]) File "/home/admin/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/predictor.py", line 89, in set_torch_image self.features = self.model.image_encoder(input_image) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/modeling/image_encoder.py", line 112, in forward x = blk(x) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/modeling/image_encoder.py", line 174, in forward x = self.attn(x) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/admin/workarea/install/segment-anything/segment_anything/modeling/image_encoder.py", line 234, in forward attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W)) File "/home/admin/workarea/install/segment-anything/segment_anything/modeling/image_encoder.py", line 358, in add_decomposed_rel_pos attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :] [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', "[>, , , , , 'CUDA out of memory. Tried to allocate 768.00 MiB (GPU 0; 10.76 GiB total capacity; 1.20 GiB already allocated; 74.88 MiB free; 1.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.013162374496459961 save_final ERROR in last step sam, CUDA out of memory. Tried to allocate 768.00 MiB (GPU 0; 10.76 GiB total capacity; 1.20 GiB already allocated; 74.88 MiB free; 1.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF time spend for datou_step_exec : 6.610702037811279 time spend to save output : 0.015717029571533203 total time spend for step 0 : 6.6264190673828125 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : None ERROR nb objects espect : 98 nb_objects detect : 0 ERROR sam FAILED ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.13533306121826172 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Tue Feb 4 18:36:36 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1738690596_598972_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1738690596_598972_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] WARNING: Logging before InitGoogleLogging() is written to STDERR F0204 18:36:39.176699 598972 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 34.08user 26.50system 1:14.41elapsed 81%CPU (0avgtext+0avgdata 3607504maxresident)k 3130512inputs+4632outputs (5988major+2942388minor)pagefaults 0swaps