python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2956 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.22644662857055664 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu Feb 27 00:35:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2956 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-02-27 00:35:31.914027: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-27 00:35:31.943046: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-27 00:35:31.945143: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff4c4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-27 00:35:31.945225: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-27 00:35:31.949705: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-27 00:35:32.220909: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x17fdb3c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-27 00:35:32.220969: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-27 00:35:32.222776: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:35:32.223573: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:35:32.228658: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:35:32.243865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:35:32.245405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:35:32.266310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:35:32.270489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:35:32.304068: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:35:32.305420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:35:32.305774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:35:32.306494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 00:35:32.306514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 00:35:32.306526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 00:35:32.308340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2504 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-02-27 00:35:32.979016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:35:32.979113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:35:32.979136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:35:32.979158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:35:32.979178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:35:32.979198: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:35:32.979229: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:35:32.979251: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:35:32.980294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:35:32.981369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:35:32.981426: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:35:32.981448: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:35:32.981468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:35:32.981487: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:35:32.981507: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:35:32.981526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:35:32.981546: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:35:32.982573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:35:32.982606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 00:35:32.982617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 00:35:32.982626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 00:35:32.983725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2504 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-27 00:35:42.111051: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:35:42.313065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:35:43.934658: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:43.934765: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:43.941472: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:43.941495: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:43.994339: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:43.994404: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:44.037091: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:44.037167: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:44.088758: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:44.088823: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:35:44.091013: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.45G (1553858560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.091565: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.30G (1398472704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.410430: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-02-27 00:35:44.457083: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.458141: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.479432: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.480424: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.481496: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.482482: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.483474: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.484083: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.498659: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.499283: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.539663: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.539738: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-27 00:35:44.540814: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.542004: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.548866: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.549460: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.557274: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.557829: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.577570: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.578332: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.579112: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.579850: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.583878: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.584470: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.585070: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.585618: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.587165: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.596634: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.597212: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.607477: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.608076: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.608673: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.609246: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.609800: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:35:44.610347: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (480, 640, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 1084966 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1763 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2956 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0008988380432128906 nb_pixel_total : 15562 time to create 1 rle with old method : 0.018377304077148438 length of segment : 256 time for calcul the mask position with numpy : 0.00274658203125 nb_pixel_total : 145328 time to create 1 rle with old method : 0.1604914665222168 length of segment : 371 time for calcul the mask position with numpy : 0.0003235340118408203 nb_pixel_total : 14255 time to create 1 rle with old method : 0.017209768295288086 length of segment : 151 time for calcul the mask position with numpy : 0.00011658668518066406 nb_pixel_total : 5613 time to create 1 rle with old method : 0.007175445556640625 length of segment : 48 time for calcul the mask position with numpy : 5.435943603515625e-05 nb_pixel_total : 1825 time to create 1 rle with old method : 0.002409696578979492 length of segment : 39 time spent for convertir_results : 0.9874081611633301 time spend for datou_step_exec : 20.93967890739441 time spend to save output : 3.123283386230469e-05 total time spend for step 1 : 20.93971014022827 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 3296 chid ids of type : 445 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.03497910499572754 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'957285035': [[(957285035, 492601069, 445, 0, 186, 22, 282, 0.99551105, [(139, 26, 7), (135, 27, 15), (133, 28, 18), (131, 29, 22), (126, 30, 28), (10, 31, 1), (120, 31, 35), (8, 32, 14), (25, 32, 6), (115, 32, 41), (7, 33, 53), (109, 33, 48), (6, 34, 71), (102, 34, 56), (5, 35, 154), (4, 36, 155), (3, 37, 156), (3, 38, 156), (3, 39, 156), (2, 40, 157), (2, 41, 157), (2, 42, 157), (2, 43, 157), (2, 44, 157), (2, 45, 157), (1, 46, 158), (1, 47, 158), (1, 48, 158), (1, 49, 157), (1, 50, 157), (1, 51, 156), (1, 52, 156), (1, 53, 155), (1, 54, 154), (1, 55, 152), (1, 56, 149), (1, 57, 145), (1, 58, 141), (1, 59, 136), (1, 60, 133), (1, 61, 130), (1, 62, 127), (1, 63, 126), (1, 64, 124), (1, 65, 123), (1, 66, 121), (1, 67, 120), (1, 68, 118), (1, 69, 117), (1, 70, 116), (1, 71, 115), (1, 72, 114), (1, 73, 113), (1, 74, 112), (1, 75, 111), (1, 76, 110), (1, 77, 108), (1, 78, 108), (1, 79, 107), (1, 80, 106), (1, 81, 105), (2, 82, 104), (2, 83, 103), (2, 84, 103), (2, 85, 102), (2, 86, 102), (2, 87, 101), (2, 88, 100), (2, 89, 99), (2, 90, 99), (2, 91, 98), (2, 92, 97), (2, 93, 96), (2, 94, 95), (2, 95, 93), (2, 96, 91), (2, 97, 90), (2, 98, 89), (2, 99, 87), (2, 100, 86), (2, 101, 86), (2, 102, 85), (2, 103, 84), (2, 104, 83), (2, 105, 83), (2, 106, 82), (2, 107, 81), (2, 108, 80), (2, 109, 80), (2, 110, 79), (2, 111, 78), (2, 112, 77), (2, 113, 76), (1, 114, 76), (1, 115, 75), (1, 116, 74), (1, 117, 73), (1, 118, 72), (1, 119, 71), (1, 120, 71), (1, 121, 70), (1, 122, 69), (1, 123, 69), (1, 124, 68), (1, 125, 68), (1, 126, 67), (1, 127, 67), (1, 128, 66), (1, 129, 66), (1, 130, 66), (1, 131, 65), (1, 132, 65), (1, 133, 64), (1, 134, 63), (1, 135, 63), (1, 136, 62), (1, 137, 61), (1, 138, 60), (1, 139, 60), (1, 140, 59), (1, 141, 58), (1, 142, 58), (1, 143, 57), (1, 144, 56), (1, 145, 56), (1, 146, 55), (1, 147, 54), (1, 148, 54), (1, 149, 53), (1, 150, 52), (1, 151, 52), (1, 152, 51), (1, 153, 50), (1, 154, 49), (1, 155, 48), (1, 156, 47), (1, 157, 46), (1, 158, 46), (1, 159, 45), (1, 160, 44), (1, 161, 43), (1, 162, 42), (1, 163, 42), (1, 164, 41), (1, 165, 40), (1, 166, 40), (1, 167, 39), (1, 168, 38), (1, 169, 37), (1, 170, 36), (1, 171, 35), (1, 172, 34), (1, 173, 34), (1, 174, 33), (1, 175, 33), (1, 176, 32), (1, 177, 32), (1, 178, 32), (1, 179, 32), (1, 180, 31), (1, 181, 31), (1, 182, 31), (1, 183, 30), (1, 184, 30), (1, 185, 30), (1, 186, 29), (1, 187, 29), (1, 188, 29), (1, 189, 28), (1, 190, 28), (1, 191, 27), (1, 192, 27), (1, 193, 26), (1, 194, 26), (1, 195, 26), (1, 196, 26), (1, 197, 26), (1, 198, 26), (1, 199, 26), (1, 200, 25), (1, 201, 25), (1, 202, 25), (1, 203, 25), (1, 204, 25), (1, 205, 25), (1, 206, 25), (1, 207, 25), (1, 208, 25), (1, 209, 25), (1, 210, 25), (1, 211, 25), (1, 212, 25), (1, 213, 25), (1, 214, 25), (1, 215, 25), (1, 216, 25), (1, 217, 25), (1, 218, 25), (1, 219, 25), (1, 220, 24), (1, 221, 24), (1, 222, 24), (1, 223, 24), (1, 224, 24), (1, 225, 24), (1, 226, 25), (1, 227, 25), (1, 228, 25), (2, 229, 24), (2, 230, 24), (2, 231, 24), (2, 232, 23), (2, 233, 23), (2, 234, 23), (2, 235, 23), (2, 236, 23), (2, 237, 23), (2, 238, 23), (2, 239, 23), (2, 240, 23), (2, 241, 23), (2, 242, 23), (2, 243, 23), (2, 244, 23), (2, 245, 23), (2, 246, 23), (2, 247, 23), (2, 248, 23), (2, 249, 24), (2, 250, 24), (2, 251, 23), (2, 252, 23), (2, 253, 23), (2, 254, 23), (2, 255, 23), (2, 256, 23), (2, 257, 23), (2, 258, 23), (2, 259, 23), (2, 260, 23), (2, 261, 23), (3, 262, 22), (3, 263, 22), (3, 264, 22), (3, 265, 22), (4, 266, 21), (4, 267, 21), (5, 268, 20), (5, 269, 20), (6, 270, 19), (7, 271, 17), (8, 272, 16), (8, 273, 16), (9, 274, 13), (11, 275, 9), (15, 276, 2)], ['16,276,8,273,2,261,2,229,1,228,1,114,2,113,2,82,1,81,1,46,3,37,8,32,30,32,31,33,59,33,60,34,76,34,77,35,101,35,114,33,120,31,130,30,135,27,145,26,152,29,158,35,158,48,154,54,141,58,128,61,119,67,105,81,103,86,96,94,89,98,81,109,71,119,65,132,60,138,52,151,42,162,40,166,34,172,32,176,29,188,26,193,25,200,25,219,24,270,23,273']), (957285035, 492601069, 445, 29, 591, 24, 419, 0.9923211, [(315, 37, 23), (272, 38, 86), (253, 39, 130), (238, 40, 151), (199, 41, 196), (189, 42, 213), (180, 43, 238), (175, 44, 250), (172, 45, 257), (169, 46, 265), (166, 47, 274), (162, 48, 284), (159, 49, 294), (157, 50, 304), (155, 51, 311), (153, 52, 317), (151, 53, 323), (150, 54, 329), (148, 55, 334), (146, 56, 337), (144, 57, 341), (142, 58, 344), (140, 59, 347), (138, 60, 350), (136, 61, 353), (134, 62, 356), (132, 63, 358), (130, 64, 361), (128, 65, 364), (126, 66, 367), (124, 67, 370), (122, 68, 373), (120, 69, 376), (118, 70, 379), (117, 71, 381), (115, 72, 385), (114, 73, 387), (113, 74, 389), (112, 75, 391), (112, 76, 393), (111, 77, 395), (110, 78, 397), (109, 79, 399), (109, 80, 400), (108, 81, 402), (107, 82, 404), (107, 83, 404), (106, 84, 406), (105, 85, 408), (105, 86, 409), (104, 87, 410), (104, 88, 411), (103, 89, 413), (102, 90, 415), (101, 91, 417), (100, 92, 420), (98, 93, 423), (97, 94, 426), (96, 95, 428), (94, 96, 431), (93, 97, 433), (92, 98, 435), (91, 99, 437), (90, 100, 439), (89, 101, 441), (89, 102, 441), (89, 103, 442), (89, 104, 443), (89, 105, 444), (89, 106, 444), (89, 107, 445), (89, 108, 446), (89, 109, 447), (89, 110, 448), (89, 111, 449), (89, 112, 450), (89, 113, 452), (89, 114, 453), (89, 115, 454), (89, 116, 455), (88, 117, 456), (88, 118, 457), (87, 119, 459), (87, 120, 459), (86, 121, 461), (86, 122, 461), (85, 123, 463), (84, 124, 464), (84, 125, 465), (83, 126, 466), (82, 127, 468), (82, 128, 468), (81, 129, 470), (80, 130, 471), (78, 131, 473), (77, 132, 475), (75, 133, 477), (73, 134, 480), (71, 135, 482), (70, 136, 484), (68, 137, 486), (67, 138, 488), (65, 139, 490), (64, 140, 492), (63, 141, 493), (61, 142, 496), (60, 143, 497), (59, 144, 499), (58, 145, 501), (58, 146, 501), (57, 147, 503), (57, 148, 504), (57, 149, 505), (56, 150, 507), (56, 151, 508), (55, 152, 509), (55, 153, 510), (54, 154, 511), (54, 155, 512), (54, 156, 513), (53, 157, 514), (53, 158, 514), (52, 159, 516), (52, 160, 516), (52, 161, 516), (51, 162, 517), (51, 163, 517), (50, 164, 518), (50, 165, 518), (49, 166, 519), (49, 167, 520), (48, 168, 521), (48, 169, 521), (47, 170, 522), (47, 171, 522), (46, 172, 523), (46, 173, 523), (46, 174, 523), (45, 175, 524), (45, 176, 523), (44, 177, 524), (44, 178, 524), (44, 179, 524), (43, 180, 525), (43, 181, 525), (42, 182, 525), (42, 183, 525), (42, 184, 525), (41, 185, 526), (41, 186, 526), (40, 187, 526), (39, 188, 526), (39, 189, 525), (38, 190, 526), (38, 191, 525), (37, 192, 525), (37, 193, 523), (36, 194, 523), (36, 195, 523), (36, 196, 522), (35, 197, 522), (35, 198, 521), (34, 199, 521), (34, 200, 521), (34, 201, 520), (34, 202, 520), (34, 203, 520), (34, 204, 519), (34, 205, 519), (33, 206, 520), (33, 207, 519), (33, 208, 519), (33, 209, 519), (33, 210, 518), (33, 211, 518), (33, 212, 518), (33, 213, 517), (32, 214, 518), (32, 215, 517), (32, 216, 517), (32, 217, 516), (32, 218, 515), (32, 219, 514), (32, 220, 513), (32, 221, 512), (32, 222, 511), (32, 223, 510), (32, 224, 508), (32, 225, 507), (32, 226, 505), (32, 227, 504), (32, 228, 503), (32, 229, 502), (32, 230, 502), (32, 231, 501), (32, 232, 500), (32, 233, 499), (32, 234, 498), (32, 235, 497), (32, 236, 495), (31, 237, 495), (31, 238, 494), (31, 239, 493), (31, 240, 491), (31, 241, 490), (31, 242, 488), (31, 243, 487), (31, 244, 486), (31, 245, 485), (31, 246, 483), (31, 247, 482), (31, 248, 480), (31, 249, 479), (31, 250, 477), (31, 251, 475), (31, 252, 474), (31, 253, 472), (31, 254, 470), (31, 255, 468), (31, 256, 467), (31, 257, 465), (31, 258, 464), (31, 259, 463), (31, 260, 462), (31, 261, 461), (31, 262, 459), (31, 263, 458), (31, 264, 456), (31, 265, 455), (31, 266, 453), (31, 267, 451), (31, 268, 449), (31, 269, 448), (31, 270, 447), (31, 271, 445), (31, 272, 444), (31, 273, 443), (32, 274, 441), (32, 275, 440), (32, 276, 438), (32, 277, 437), (32, 278, 435), (32, 279, 434), (32, 280, 432), (33, 281, 429), (33, 282, 427), (33, 283, 426), (33, 284, 424), (33, 285, 423), (34, 286, 421), (34, 287, 420), (34, 288, 419), (35, 289, 416), (35, 290, 415), (35, 291, 414), (36, 292, 411), (36, 293, 410), (37, 294, 407), (37, 295, 406), (38, 296, 403), (38, 297, 401), (39, 298, 399), (39, 299, 397), (41, 300, 394), (42, 301, 392), (43, 302, 389), (44, 303, 387), (45, 304, 385), (46, 305, 382), (47, 306, 380), (47, 307, 378), (48, 308, 376), (49, 309, 373), (50, 310, 370), (51, 311, 368), (51, 312, 367), (52, 313, 365), (54, 314, 362), (55, 315, 360), (56, 316, 359), (58, 317, 356), (61, 318, 352), (64, 319, 349), (67, 320, 345), (70, 321, 341), (73, 322, 338), (75, 323, 335), (78, 324, 332), (80, 325, 329), (82, 326, 327), (84, 327, 324), (86, 328, 322), (88, 329, 320), (90, 330, 317), (93, 331, 314), (96, 332, 311), (99, 333, 307), (102, 334, 304), (105, 335, 300), (108, 336, 297), (111, 337, 294), (113, 338, 291), (115, 339, 289), (117, 340, 286), (119, 341, 283), (121, 342, 281), (123, 343, 278), (125, 344, 275), (127, 345, 272), (129, 346, 269), (132, 347, 266), (135, 348, 262), (138, 349, 258), (141, 350, 255), (143, 351, 252), (145, 352, 250), (147, 353, 247), (149, 354, 245), (151, 355, 242), (152, 356, 241), (154, 357, 239), (156, 358, 237), (159, 359, 233), (161, 360, 231), (163, 361, 229), (165, 362, 227), (167, 363, 224), (169, 364, 222), (170, 365, 221), (172, 366, 219), (173, 367, 218), (174, 368, 216), (175, 369, 215), (177, 370, 213), (178, 371, 212), (180, 372, 209), (183, 373, 206), (185, 374, 204), (188, 375, 200), (191, 376, 197), (194, 377, 193), (196, 378, 191), (199, 379, 188), (201, 380, 185), (203, 381, 183), (205, 382, 180), (207, 383, 178), (208, 384, 176), (210, 385, 174), (212, 386, 171), (213, 387, 169), (216, 388, 165), (218, 389, 162), (221, 390, 158), (225, 391, 153), (228, 392, 149), (232, 393, 144), (235, 394, 140), (238, 395, 136), (241, 396, 133), (245, 397, 128), (248, 398, 124), (252, 399, 119), (256, 400, 114), (263, 401, 105), (272, 402, 94), (283, 403, 81), (296, 404, 66), (306, 405, 53), (313, 406, 38), (321, 407, 23)], ['321,407,305,404,263,401,218,389,193,376,178,371,168,363,140,349,110,336,90,330,77,323,56,316,39,299,31,273,31,237,34,199,58,145,82,128,89,116,89,101,104,88,115,72,159,49,180,43,199,41,237,41,272,38,337,37,382,39,402,43,460,50,481,55,529,101,543,116,556,143,567,159,566,186,554,199,548,216,528,235,496,256,471,275,460,281,414,315,403,339,392,355,389,371,383,385,369,400,358,405']), (957285035, 492601069, 445, 485, 636, 23, 174, 0.97113144, [(540, 24, 21), (626, 24, 3), (531, 25, 49), (594, 25, 40), (527, 26, 107), (523, 27, 111), (520, 28, 114), (517, 29, 118), (516, 30, 119), (515, 31, 120), (513, 32, 122), (512, 33, 123), (510, 34, 125), (509, 35, 126), (507, 36, 128), (506, 37, 129), (504, 38, 131), (503, 39, 132), (501, 40, 134), (500, 41, 135), (499, 42, 136), (498, 43, 137), (497, 44, 138), (496, 45, 139), (496, 46, 139), (495, 47, 140), (495, 48, 140), (494, 49, 141), (493, 50, 142), (492, 51, 143), (491, 52, 144), (491, 53, 144), (490, 54, 145), (490, 55, 145), (490, 56, 145), (490, 57, 146), (490, 58, 146), (490, 59, 146), (491, 60, 145), (491, 61, 145), (491, 62, 145), (492, 63, 144), (493, 64, 143), (494, 65, 142), (495, 66, 141), (496, 67, 140), (497, 68, 138), (498, 69, 138), (499, 70, 137), (500, 71, 136), (501, 72, 135), (503, 73, 133), (503, 74, 133), (505, 75, 131), (506, 76, 130), (507, 77, 129), (508, 78, 128), (509, 79, 127), (510, 80, 126), (511, 81, 125), (512, 82, 124), (513, 83, 123), (514, 84, 122), (515, 85, 121), (516, 86, 120), (517, 87, 119), (518, 88, 118), (519, 89, 117), (521, 90, 115), (521, 91, 115), (522, 92, 114), (523, 93, 113), (524, 94, 112), (525, 95, 111), (526, 96, 110), (527, 97, 109), (529, 98, 107), (530, 99, 106), (532, 100, 104), (533, 101, 103), (534, 102, 102), (535, 103, 101), (536, 104, 100), (538, 105, 98), (540, 106, 96), (541, 107, 95), (543, 108, 93), (546, 109, 90), (548, 110, 88), (549, 111, 87), (551, 112, 84), (552, 113, 83), (553, 114, 82), (555, 115, 80), (556, 116, 79), (556, 117, 79), (557, 118, 78), (558, 119, 77), (559, 120, 76), (560, 121, 75), (560, 122, 75), (561, 123, 74), (561, 124, 74), (561, 125, 74), (562, 126, 73), (562, 127, 73), (563, 128, 72), (563, 129, 72), (564, 130, 70), (564, 131, 70), (565, 132, 69), (565, 133, 68), (565, 134, 68), (565, 135, 67), (566, 136, 65), (566, 137, 64), (566, 138, 64), (566, 139, 62), (566, 140, 61), (566, 141, 59), (566, 142, 57), (566, 143, 56), (566, 144, 55), (566, 145, 54), (567, 146, 53), (567, 147, 52), (567, 148, 51), (568, 149, 50), (568, 150, 49), (568, 151, 48), (568, 152, 47), (569, 153, 45), (569, 154, 44), (570, 155, 42), (570, 156, 42), (570, 157, 41), (571, 158, 39), (571, 159, 39), (572, 160, 37), (572, 161, 37), (573, 162, 35), (573, 163, 34), (573, 164, 34), (574, 165, 32), (575, 166, 30), (577, 167, 28), (578, 168, 26), (581, 169, 22), (584, 170, 19), (587, 171, 15), (591, 172, 8)], ['598,172,591,172,590,171,578,168,573,164,573,162,568,152,568,149,566,145,566,136,565,132,561,125,560,121,556,116,547,109,543,108,536,104,531,99,527,97,491,62,490,54,495,48,496,45,501,40,514,32,517,29,531,25,539,25,540,24,560,24,561,25,579,25,580,26,593,26,594,25,633,25,634,29,634,56,635,57,635,111,634,112,634,129,632,134,629,138,623,141,619,145,617,149,611,155,608,161,604,166']), (957285035, 492601069, 445, 280, 481, 2, 55, 0.83013344, [(292, 3, 128), (284, 4, 146), (282, 5, 151), (281, 6, 154), (281, 7, 156), (281, 8, 157), (281, 9, 158), (281, 10, 160), (281, 11, 162), (281, 12, 165), (281, 13, 167), (281, 14, 169), (281, 15, 171), (281, 16, 173), (281, 17, 174), (281, 18, 175), (281, 19, 177), (281, 20, 178), (281, 21, 179), (281, 22, 180), (281, 23, 181), (281, 24, 182), (281, 25, 183), (281, 26, 184), (281, 27, 185), (281, 28, 185), (281, 29, 185), (282, 30, 185), (283, 31, 27), (337, 31, 131), (371, 32, 97), (401, 33, 68), (409, 34, 61), (419, 35, 52), (424, 36, 48), (429, 37, 44), (432, 38, 41), (434, 39, 40), (436, 40, 39), (438, 41, 37), (441, 42, 35), (444, 43, 32), (448, 44, 29), (452, 45, 25), (454, 46, 23), (459, 47, 17), (463, 48, 12), (468, 49, 5)], ['472,49,468,49,467,48,459,47,458,46,454,46,451,44,448,44,447,43,444,43,440,41,438,41,428,36,424,36,423,35,419,35,418,34,409,34,408,33,401,33,400,32,371,32,370,31,337,31,336,30,283,31,281,29,281,6,284,4,291,4,292,3,419,3,420,4,429,4,430,5,432,5,436,7,441,11,445,12,453,16,456,19,457,19,465,27,465,29,472,37,476,44,476,46']), (957285035, 492601069, 445, 456, 547, 6, 45, 0.7411352, [(482, 8, 19), (463, 9, 4), (481, 9, 44), (457, 10, 12), (479, 10, 50), (457, 11, 13), (476, 11, 56), (457, 12, 15), (475, 12, 65), (457, 13, 84), (457, 14, 85), (457, 15, 89), (457, 16, 89), (458, 17, 88), (459, 18, 87), (460, 19, 86), (461, 20, 80), (464, 21, 71), (466, 22, 63), (467, 23, 59), (468, 24, 55), (469, 25, 52), (469, 26, 51), (470, 27, 48), (471, 28, 46), (471, 29, 44), (472, 30, 42), (473, 31, 39), (473, 32, 38), (474, 33, 36), (475, 34, 33), (475, 35, 32), (476, 36, 30), (476, 37, 29), (477, 38, 26), (478, 39, 23), (479, 40, 20), (480, 41, 17), (488, 42, 5)], ['492,42,488,42,487,41,480,41,476,37,475,34,473,32,469,25,465,21,461,20,457,16,457,10,466,9,470,12,474,13,476,11,480,10,482,8,500,8,501,9,524,9,525,10,528,10,532,12,539,12,542,15,545,15,545,19,535,20,534,21,529,21,525,23,523,23,513,30,512,30,504,37,496,41,493,41'])], 'temp/1740612927_1084700_957285035_a42482e51c93c8025d243dd179aee85b.jpg']} free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2956 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.17197751998901367 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu Feb 27 00:35:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2956 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-27 00:35:54.473789: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-27 00:35:54.503172: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-27 00:35:54.505447: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff4c8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-27 00:35:54.505518: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-27 00:35:54.509630: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-27 00:35:54.777821: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x189b4a40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-27 00:35:54.777866: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-27 00:35:54.778400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:35:54.778730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:35:54.780782: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:35:54.782946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:35:54.783266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:35:54.789324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:35:54.791000: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:35:54.797871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:35:54.799221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:35:54.799320: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:35:54.800097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 00:35:54.800118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 00:35:54.800129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 00:35:54.801293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2504 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-27 00:35:54.881404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:35:54.881531: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:35:54.881551: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:35:54.881567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:35:54.881584: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:35:54.881599: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:35:54.881615: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:35:54.881633: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:35:54.882419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:35:54.883393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:35:54.883432: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:35:54.883450: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:35:54.883466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:35:54.883481: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:35:54.883497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:35:54.883513: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:35:54.883529: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:35:54.884494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:35:54.884532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 00:35:54.884542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 00:35:54.884552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 00:35:54.885589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2504 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-27 00:36:02.648951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:36:02.870036: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:36:04.453722: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.453788: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.463152: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.463213: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.528113: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.528181: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.578363: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.578410: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.629970: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.630017: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-27 00:36:04.633089: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.45G (1553858560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.633584: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.30G (1398472704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.634052: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.17G (1258625536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.634064: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-02-27 00:36:04.648186: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.649181: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.657230: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.657815: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.662836: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.663434: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.677298: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.677845: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.679343: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.679924: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.686535: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.687108: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.690571: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.691159: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.697402: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.697982: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.700072: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.700636: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.735210: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.735843: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.736427: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.737009: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.740984: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.741606: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.760484: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.761113: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.761697: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.762415: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.777572: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.778179: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.778937: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.779524: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.784849: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.785485: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.791999: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.792634: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.806081: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.806720: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.811822: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.812437: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.812989: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.813526: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.836372: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.837018: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.837623: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.838194: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.838772: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.839378: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.858594: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.859726: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.887212: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.887305: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-27 00:36:04.888394: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.889469: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.897726: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.898520: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.908064: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.908662: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.926057: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.926790: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.927472: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.928155: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.933496: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.934629: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.935706: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.936744: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.938689: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.948339: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.949059: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.959529: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.960162: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.960766: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.961406: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.962034: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:36:04.962633: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2090729472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (720, 1280, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 1280.00000 nb d'objets trouves : 4 Detection mask done ! Trying to reset tf kernel 1086234 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 964 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2157 list_Values should be empty [] ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0007572174072265625 nb_pixel_total : 16902 time to create 1 rle with old method : 0.025293588638305664 length of segment : 107 time for calcul the mask position with numpy : 0.02689218521118164 nb_pixel_total : 480745 time to create 1 rle with new method : 0.2303929328918457 length of segment : 632 time for calcul the mask position with numpy : 0.0005214214324951172 nb_pixel_total : 36581 time to create 1 rle with old method : 0.04210376739501953 length of segment : 132 time for calcul the mask position with numpy : 0.00011277198791503906 nb_pixel_total : 4793 time to create 1 rle with old method : 0.005430936813354492 length of segment : 51 time spent for convertir_results : 0.5162222385406494 time spend for datou_step_exec : 16.934722900390625 time spend to save output : 4.649162292480469e-05 total time spend for step 1 : 16.93476939201355 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 400 chid ids of type : 445 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.03746151924133301 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'917855882': [[(917855882, 492601069, 445, 1092, 1280, 0, 108, 0.99883634, [(1205, 1, 58), (1165, 2, 105), (1159, 3, 113), (1149, 4, 124), (1113, 5, 161), (1100, 6, 174), (1097, 7, 177), (1095, 8, 179), (1095, 9, 179), (1095, 10, 179), (1095, 11, 179), (1095, 12, 179), (1095, 13, 179), (1095, 14, 178), (1095, 15, 178), (1095, 16, 178), (1095, 17, 178), (1095, 18, 177), (1095, 19, 177), (1095, 20, 177), (1095, 21, 177), (1095, 22, 177), (1095, 23, 178), (1095, 24, 178), (1095, 25, 178), (1095, 26, 179), (1095, 27, 179), (1095, 28, 180), (1095, 29, 181), (1095, 30, 182), (1095, 31, 183), (1095, 32, 183), (1095, 33, 184), (1095, 34, 184), (1096, 35, 183), (1096, 36, 183), (1096, 37, 184), (1097, 38, 183), (1097, 39, 183), (1097, 40, 183), (1098, 41, 182), (1098, 42, 182), (1098, 43, 182), (1099, 44, 181), (1099, 45, 181), (1099, 46, 181), (1100, 47, 180), (1100, 48, 180), (1101, 49, 179), (1101, 50, 179), (1102, 51, 178), (1102, 52, 178), (1103, 53, 177), (1103, 54, 177), (1104, 55, 176), (1104, 56, 176), (1104, 57, 176), (1104, 58, 176), (1105, 59, 175), (1105, 60, 175), (1105, 61, 175), (1105, 62, 175), (1105, 63, 175), (1106, 64, 174), (1106, 65, 174), (1106, 66, 174), (1106, 67, 174), (1106, 68, 174), (1106, 69, 174), (1106, 70, 174), (1106, 71, 174), (1106, 72, 174), (1106, 73, 174), (1107, 74, 173), (1107, 75, 173), (1107, 76, 173), (1107, 77, 173), (1107, 78, 173), (1107, 79, 173), (1108, 80, 172), (1108, 81, 172), (1109, 82, 171), (1110, 83, 170), (1110, 84, 170), (1111, 85, 169), (1112, 86, 168), (1113, 87, 166), (1114, 88, 165), (1115, 89, 164), (1117, 90, 162), (1120, 91, 159), (1138, 92, 141), (1146, 93, 133), (1154, 94, 125), (1167, 95, 112), (1177, 96, 102), (1183, 97, 95), (1185, 98, 93), (1187, 99, 90), (1188, 100, 55), (1264, 100, 12), (1190, 101, 50), (1191, 102, 46), (1194, 103, 40), (1197, 104, 34), (1202, 105, 25), (1207, 106, 16)], ['1222,106,1207,106,1206,105,1197,104,1191,102,1182,96,1176,95,1167,95,1166,94,1154,94,1153,93,1146,93,1145,92,1137,91,1120,91,1115,89,1110,84,1107,79,1106,73,1106,64,1104,55,1099,46,1095,34,1095,8,1100,6,1112,6,1113,5,1148,5,1149,4,1158,4,1165,2,1204,2,1205,1,1262,1,1269,2,1273,5,1273,13,1271,18,1271,22,1273,27,1277,31,1279,37,1279,86,1278,87,1278,96,1275,100,1264,100,1263,99,1243,99,1230,104']), (917855882, 492601069, 445, 52, 1128, 16, 668, 0.9977391, [(711, 22, 21), (926, 22, 46), (608, 23, 146), (894, 23, 103), (598, 24, 233), (850, 24, 158), (590, 25, 427), (582, 26, 444), (575, 27, 458), (569, 28, 466), (565, 29, 472), (560, 30, 480), (556, 31, 486), (550, 32, 495), (545, 33, 502), (538, 34, 512), (532, 35, 520), (527, 36, 527), (523, 37, 534), (518, 38, 541), (514, 39, 548), (510, 40, 554), (506, 41, 561), (503, 42, 566), (499, 43, 572), (496, 44, 577), (493, 45, 582), (491, 46, 585), (488, 47, 590), (487, 48, 592), (485, 49, 595), (483, 50, 598), (482, 51, 600), (481, 52, 602), (480, 53, 603), (479, 54, 605), (478, 55, 606), (476, 56, 608), (475, 57, 610), (474, 58, 611), (473, 59, 613), (472, 60, 614), (470, 61, 616), (469, 62, 618), (468, 63, 619), (466, 64, 621), (465, 65, 623), (464, 66, 624), (462, 67, 626), (461, 68, 628), (459, 69, 630), (458, 70, 631), (456, 71, 633), (455, 72, 635), (453, 73, 637), (452, 74, 638), (451, 75, 639), (450, 76, 640), (448, 77, 642), (447, 78, 643), (446, 79, 644), (445, 80, 645), (444, 81, 646), (442, 82, 648), (441, 83, 649), (440, 84, 650), (439, 85, 651), (438, 86, 652), (437, 87, 653), (436, 88, 654), (435, 89, 655), (434, 90, 656), (433, 91, 657), (432, 92, 658), (431, 93, 659), (430, 94, 660), (429, 95, 661), (428, 96, 662), (427, 97, 663), (425, 98, 665), (423, 99, 667), (421, 100, 669), (419, 101, 671), (417, 102, 673), (413, 103, 677), (410, 104, 680), (405, 105, 685), (401, 106, 689), (397, 107, 693), (392, 108, 698), (387, 109, 703), (382, 110, 708), (377, 111, 713), (373, 112, 717), (368, 113, 722), (365, 114, 725), (361, 115, 729), (358, 116, 732), (356, 117, 734), (353, 118, 737), (351, 119, 739), (348, 120, 742), (346, 121, 744), (344, 122, 746), (341, 123, 749), (338, 124, 752), (335, 125, 755), (331, 126, 759), (327, 127, 763), (323, 128, 767), (319, 129, 770), (314, 130, 775), (308, 131, 781), (303, 132, 786), (294, 133, 795), (286, 134, 803), (279, 135, 810), (273, 136, 816), (266, 137, 823), (262, 138, 827), (258, 139, 831), (255, 140, 834), (252, 141, 837), (250, 142, 839), (247, 143, 842), (245, 144, 844), (242, 145, 847), (240, 146, 849), (237, 147, 852), (233, 148, 856), (230, 149, 859), (226, 150, 863), (220, 151, 869), (213, 152, 876), (206, 153, 883), (200, 154, 889), (193, 155, 896), (187, 156, 902), (183, 157, 906), (181, 158, 908), (178, 159, 911), (176, 160, 913), (174, 161, 915), (172, 162, 917), (170, 163, 919), (168, 164, 921), (167, 165, 922), (165, 166, 924), (164, 167, 925), (162, 168, 927), (161, 169, 928), (159, 170, 930), (157, 171, 932), (155, 172, 934), (153, 173, 935), (151, 174, 937), (148, 175, 940), (146, 176, 942), (144, 177, 944), (142, 178, 946), (140, 179, 948), (139, 180, 949), (137, 181, 951), (136, 182, 952), (134, 183, 954), (133, 184, 955), (132, 185, 956), (131, 186, 957), (130, 187, 958), (129, 188, 959), (128, 189, 960), (127, 190, 960), (126, 191, 961), (126, 192, 961), (125, 193, 962), (124, 194, 963), (123, 195, 964), (122, 196, 965), (122, 197, 965), (121, 198, 966), (120, 199, 967), (119, 200, 968), (118, 201, 969), (117, 202, 970), (116, 203, 971), (114, 204, 973), (113, 205, 973), (112, 206, 974), (111, 207, 975), (109, 208, 977), (108, 209, 978), (107, 210, 979), (106, 211, 980), (105, 212, 981), (104, 213, 982), (103, 214, 983), (102, 215, 984), (101, 216, 985), (101, 217, 984), (100, 218, 985), (99, 219, 986), (99, 220, 986), (98, 221, 987), (98, 222, 987), (97, 223, 988), (97, 224, 987), (96, 225, 988), (96, 226, 988), (95, 227, 989), (95, 228, 989), (94, 229, 990), (94, 230, 990), (94, 231, 990), (93, 232, 990), (93, 233, 990), (92, 234, 991), (92, 235, 991), (92, 236, 991), (91, 237, 992), (91, 238, 991), (91, 239, 991), (91, 240, 990), (91, 241, 990), (90, 242, 991), (90, 243, 990), (90, 244, 990), (90, 245, 989), (90, 246, 989), (89, 247, 990), (89, 248, 989), (89, 249, 989), (89, 250, 988), (89, 251, 988), (88, 252, 988), (88, 253, 988), (88, 254, 987), (88, 255, 986), (88, 256, 986), (87, 257, 986), (87, 258, 985), (87, 259, 985), (87, 260, 984), (87, 261, 983), (86, 262, 983), (86, 263, 982), (86, 264, 982), (86, 265, 981), (85, 266, 981), (85, 267, 980), (85, 268, 980), (84, 269, 980), (84, 270, 979), (84, 271, 979), (84, 272, 978), (83, 273, 979), (83, 274, 978), (83, 275, 977), (82, 276, 978), (82, 277, 977), (82, 278, 977), (81, 279, 977), (81, 280, 977), (81, 281, 977), (80, 282, 977), (80, 283, 977), (80, 284, 976), (79, 285, 977), (79, 286, 976), (79, 287, 976), (78, 288, 976), (78, 289, 976), (78, 290, 975), (77, 291, 976), (77, 292, 975), (77, 293, 975), (76, 294, 975), (76, 295, 975), (76, 296, 974), (75, 297, 975), (75, 298, 974), (74, 299, 975), (74, 300, 974), (74, 301, 974), (73, 302, 974), (73, 303, 974), (72, 304, 974), (72, 305, 974), (71, 306, 974), (71, 307, 973), (71, 308, 972), (70, 309, 972), (70, 310, 971), (70, 311, 970), (70, 312, 968), (69, 313, 968), (69, 314, 966), (69, 315, 964), (69, 316, 962), (68, 317, 961), (68, 318, 959), (68, 319, 958), (68, 320, 956), (67, 321, 955), (67, 322, 954), (67, 323, 952), (67, 324, 951), (66, 325, 951), (66, 326, 950), (66, 327, 948), (66, 328, 947), (65, 329, 947), (65, 330, 946), (65, 331, 946), (65, 332, 945), (65, 333, 944), (65, 334, 942), (65, 335, 941), (65, 336, 940), (65, 337, 939), (65, 338, 938), (65, 339, 936), (64, 340, 936), (64, 341, 934), (64, 342, 932), (64, 343, 930), (64, 344, 928), (64, 345, 927), (64, 346, 925), (64, 347, 923), (64, 348, 922), (64, 349, 920), (64, 350, 919), (63, 351, 919), (63, 352, 918), (63, 353, 917), (63, 354, 916), (63, 355, 915), (63, 356, 914), (63, 357, 912), (63, 358, 911), (63, 359, 910), (63, 360, 909), (63, 361, 908), (63, 362, 906), (63, 363, 905), (63, 364, 904), (63, 365, 902), (63, 366, 901), (63, 367, 899), (63, 368, 898), (63, 369, 896), (62, 370, 895), (62, 371, 893), (62, 372, 891), (62, 373, 890), (62, 374, 888), (62, 375, 887), (62, 376, 886), (62, 377, 885), (62, 378, 884), (62, 379, 883), (63, 380, 880), (63, 381, 879), (63, 382, 878), (63, 383, 877), (63, 384, 876), (63, 385, 875), (63, 386, 874), (63, 387, 873), (64, 388, 871), (64, 389, 870), (64, 390, 869), (64, 391, 868), (64, 392, 867), (64, 393, 866), (64, 394, 864), (64, 395, 863), (65, 396, 861), (65, 397, 860), (65, 398, 859), (65, 399, 858), (65, 400, 857), (65, 401, 856), (65, 402, 854), (65, 403, 853), (65, 404, 851), (65, 405, 850), (65, 406, 848), (66, 407, 846), (66, 408, 844), (66, 409, 843), (66, 410, 842), (66, 411, 841), (66, 412, 840), (66, 413, 838), (66, 414, 837), (66, 415, 836), (66, 416, 835), (66, 417, 835), (66, 418, 834), (66, 419, 833), (67, 420, 831), (67, 421, 830), (67, 422, 829), (67, 423, 829), (67, 424, 828), (67, 425, 827), (67, 426, 826), (67, 427, 825), (67, 428, 824), (68, 429, 822), (68, 430, 820), (68, 431, 819), (68, 432, 818), (68, 433, 816), (68, 434, 815), (68, 435, 813), (68, 436, 811), (69, 437, 809), (69, 438, 807), (69, 439, 806), (69, 440, 804), (69, 441, 803), (69, 442, 802), (69, 443, 800), (70, 444, 798), (70, 445, 797), (70, 446, 796), (70, 447, 796), (71, 448, 794), (71, 449, 794), (72, 450, 792), (72, 451, 792), (73, 452, 790), (73, 453, 789), (74, 454, 788), (74, 455, 787), (75, 456, 786), (75, 457, 785), (76, 458, 784), (76, 459, 783), (77, 460, 782), (77, 461, 781), (77, 462, 781), (78, 463, 779), (78, 464, 779), (79, 465, 777), (79, 466, 777), (79, 467, 776), (80, 468, 775), (80, 469, 774), (80, 470, 774), (81, 471, 772), (81, 472, 771), (82, 473, 770), (82, 474, 769), (83, 475, 767), (83, 476, 766), (83, 477, 766), (84, 478, 764), (84, 479, 763), (85, 480, 761), (85, 481, 760), (85, 482, 759), (86, 483, 757), (86, 484, 755), (87, 485, 753), (87, 486, 752), (87, 487, 751), (88, 488, 748), (88, 489, 747), (88, 490, 746), (89, 491, 744), (89, 492, 743), (90, 493, 741), (90, 494, 741), (91, 495, 739), (91, 496, 738), (92, 497, 737), (93, 498, 735), (94, 499, 733), (94, 500, 733), (95, 501, 731), (96, 502, 729), (97, 503, 728), (98, 504, 726), (99, 505, 724), (99, 506, 724), (100, 507, 722), (101, 508, 721), (102, 509, 719), (104, 510, 717), (105, 511, 715), (106, 512, 714), (107, 513, 712), (108, 514, 711), (110, 515, 708), (111, 516, 707), (113, 517, 704), (114, 518, 703), (115, 519, 701), (117, 520, 698), (118, 521, 697), (119, 522, 695), (121, 523, 693), (122, 524, 691), (124, 525, 689), (125, 526, 687), (126, 527, 685), (128, 528, 683), (129, 529, 681), (131, 530, 678), (132, 531, 676), (134, 532, 674), (135, 533, 672), (137, 534, 669), (138, 535, 667), (140, 536, 664), (141, 537, 662), (143, 538, 659), (144, 539, 657), (146, 540, 654), (148, 541, 651), (149, 542, 649), (151, 543, 645), (153, 544, 642), (154, 545, 641), (156, 546, 638), (158, 547, 635), (159, 548, 633), (161, 549, 630), (162, 550, 628), (164, 551, 626), (166, 552, 623), (167, 553, 621), (169, 554, 618), (170, 555, 617), (171, 556, 615), (173, 557, 613), (174, 558, 611), (176, 559, 608), (177, 560, 607), (178, 561, 605), (180, 562, 603), (181, 563, 601), (183, 564, 599), (185, 565, 597), (186, 566, 595), (189, 567, 592), (192, 568, 589), (195, 569, 585), (198, 570, 582), (201, 571, 579), (204, 572, 575), (206, 573, 573), (209, 574, 569), (212, 575, 566), (215, 576, 563), (218, 577, 559), (221, 578, 556), (223, 579, 553), (226, 580, 550), (228, 581, 547), (230, 582, 545), (232, 583, 542), (234, 584, 540), (235, 585, 539), (237, 586, 536), (238, 587, 534), (240, 588, 531), (242, 589, 528), (243, 590, 526), (245, 591, 523), (247, 592, 520), (249, 593, 516), (251, 594, 513), (253, 595, 510), (256, 596, 505), (258, 597, 501), (261, 598, 497), (263, 599, 493), (267, 600, 488), (271, 601, 482), (274, 602, 478), (278, 603, 473), (281, 604, 468), (284, 605, 464), (287, 606, 460), (290, 607, 456), (292, 608, 453), (295, 609, 449), (298, 610, 445), (300, 611, 442), (303, 612, 438), (305, 613, 434), (308, 614, 430), (310, 615, 427), (312, 616, 423), (315, 617, 418), (317, 618, 415), (320, 619, 410), (322, 620, 406), (325, 621, 401), (328, 622, 395), (330, 623, 390), (333, 624, 384), (335, 625, 379), (338, 626, 374), (341, 627, 369), (345, 628, 362), (349, 629, 356), (353, 630, 350), (357, 631, 344), (360, 632, 340), (364, 633, 334), (368, 634, 327), (373, 635, 320), (378, 636, 313), (384, 637, 304), (389, 638, 294), (395, 639, 282), (401, 640, 270), (408, 641, 256), (416, 642, 240), (432, 643, 216), (449, 644, 192), (465, 645, 169), (480, 646, 148), (495, 647, 126), (511, 648, 104), (526, 649, 82), (565, 650, 9)], ['526,649,416,642,341,627,243,590,220,577,186,566,119,522,91,496,70,447,63,380,65,329,86,265,91,237,101,216,134,183,187,156,225,151,266,137,302,133,343,123,358,116,416,103,448,77,493,45,527,36,608,23,753,23,754,24,893,24,926,22,996,23,1032,27,1066,41,1082,52,1089,72,1088,172,1082,237,1045,305,1002,338,950,373,885,432,865,446,851,473,822,505,810,528,786,554,773,585,714,624,687,637,607,649']), (917855882, 492601069, 445, 0, 438, 0, 116, 0.99194664, [(127, 1, 140), (94, 2, 205), (59, 3, 273), (338, 3, 59), (22, 4, 380), (19, 5, 386), (16, 6, 391), (15, 7, 393), (14, 8, 395), (14, 9, 396), (13, 10, 398), (12, 11, 399), (12, 12, 399), (11, 13, 401), (10, 14, 402), (11, 15, 402), (11, 16, 403), (12, 17, 403), (12, 18, 404), (12, 19, 405), (12, 20, 405), (12, 21, 405), (12, 22, 406), (12, 23, 406), (12, 24, 406), (12, 25, 407), (12, 26, 407), (12, 27, 407), (12, 28, 408), (12, 29, 408), (12, 30, 408), (12, 31, 408), (12, 32, 408), (12, 33, 408), (12, 34, 409), (12, 35, 409), (12, 36, 409), (12, 37, 409), (12, 38, 409), (12, 39, 409), (12, 40, 409), (12, 41, 409), (12, 42, 409), (12, 43, 410), (12, 44, 410), (12, 45, 409), (12, 46, 409), (12, 47, 409), (12, 48, 409), (12, 49, 409), (12, 50, 409), (12, 51, 409), (12, 52, 408), (12, 53, 408), (12, 54, 407), (12, 55, 406), (12, 56, 405), (12, 57, 404), (12, 58, 402), (11, 59, 402), (11, 60, 401), (11, 61, 401), (11, 62, 400), (11, 63, 399), (11, 64, 399), (11, 65, 398), (11, 66, 397), (11, 67, 396), (11, 68, 395), (11, 69, 395), (11, 70, 394), (11, 71, 394), (11, 72, 394), (11, 73, 393), (11, 74, 393), (11, 75, 393), (11, 76, 393), (11, 77, 393), (11, 78, 393), (11, 79, 392), (11, 80, 392), (10, 81, 393), (10, 82, 394), (10, 83, 394), (9, 84, 396), (9, 85, 263), (284, 85, 121), (9, 86, 75), (97, 86, 29), (141, 86, 120), (292, 86, 112), (9, 87, 71), (152, 87, 104), (294, 87, 110), (8, 88, 67), (160, 88, 92), (295, 88, 108), (8, 89, 63), (175, 89, 73), (296, 89, 107), (7, 90, 61), (204, 90, 41), (297, 90, 105), (7, 91, 57), (298, 91, 104), (7, 92, 53), (299, 92, 103), (6, 93, 50), (300, 93, 101), (7, 94, 46), (303, 94, 96), (7, 95, 44), (305, 95, 93), (7, 96, 42), (308, 96, 88), (7, 97, 40), (310, 97, 85), (7, 98, 38), (312, 98, 82), (8, 99, 34), (314, 99, 79), (8, 100, 32), (316, 100, 75), (8, 101, 29), (319, 101, 71), (13, 102, 18), (325, 102, 62), (20, 103, 6), (331, 103, 51), (337, 104, 37), (344, 105, 22), (351, 106, 2)], ['344,105,319,101,307,95,300,93,283,84,261,85,244,90,204,90,203,89,175,89,160,88,140,85,125,86,97,86,84,85,67,90,56,92,36,101,25,103,8,101,6,93,11,80,11,59,12,58,12,17,10,14,16,6,22,4,58,4,59,3,93,3,94,2,126,2,127,1,266,1,267,2,298,2,299,3,396,3,406,6,416,19,420,34,420,51,411,60,405,68,402,81,404,85,401,92,386,102,365,105']), (917855882, 492601069, 445, 390, 550, 0, 54, 0.9391322, [(414, 0, 7), (441, 0, 60), (508, 0, 28), (402, 1, 142), (401, 2, 146), (402, 3, 145), (404, 4, 143), (406, 5, 140), (408, 6, 137), (410, 7, 134), (411, 8, 132), (412, 9, 130), (413, 10, 127), (414, 11, 125), (415, 12, 123), (415, 13, 122), (416, 14, 120), (417, 15, 117), (417, 16, 116), (418, 17, 114), (418, 18, 113), (418, 19, 111), (418, 20, 109), (419, 21, 107), (419, 22, 105), (419, 23, 103), (419, 24, 102), (420, 25, 99), (420, 26, 97), (420, 27, 95), (420, 28, 94), (421, 29, 91), (421, 30, 90), (422, 31, 88), (422, 32, 88), (422, 33, 87), (423, 34, 84), (423, 35, 82), (423, 36, 81), (424, 37, 79), (424, 38, 77), (424, 39, 75), (424, 40, 73), (424, 41, 71), (425, 42, 67), (425, 43, 66), (426, 44, 62), (426, 45, 6), (433, 45, 52), (443, 46, 30), (450, 47, 1)], ['449,46,443,46,442,45,426,45,424,41,424,37,423,36,422,31,420,28,420,25,419,24,419,21,418,20,418,17,417,15,409,6,402,3,402,1,413,1,414,0,420,0,421,1,440,1,441,0,500,0,501,1,507,1,508,0,535,0,536,1,543,1,546,2,546,4,542,8,530,18,527,19,525,21,522,22,520,24,512,28,508,33,505,34,502,37,494,41,492,41,490,43,488,43,484,45,473,45,472,46'])], 'temp/1740612951_1084700_917855882_da0fa7b7e6b5b551fe26c0ba8713276d.jpg']} ############################### TEST POLYGON ################################ Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.17495417594909668 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu Feb 27 00:36:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 493 wait 20 seconds l 3637 free memory gpu now : 493 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-27 00:36:51.433415: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-27 00:36:51.463190: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-27 00:36:51.465440: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff4c8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-27 00:36:51.465507: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-27 00:36:51.469498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-27 00:36:51.627264: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x193b34f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-27 00:36:51.627324: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-27 00:36:51.628272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:36:51.628682: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:36:51.631470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:36:51.634097: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:36:51.634583: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:36:51.637419: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:36:51.638454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:36:51.642411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:36:51.643358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:36:51.643440: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:36:51.643929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 00:36:51.643944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 00:36:51.643953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 00:36:51.644754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 116 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-27 00:36:51.730285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:36:51.730389: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:36:51.730421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:36:51.730449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:36:51.730478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:36:51.730506: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:36:51.730534: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:36:51.730562: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:36:51.732151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:36:51.733692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:36:51.733815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:36:51.733853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:36:51.733887: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:36:51.733921: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:36:51.733976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:36:51.734015: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:36:51.734050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:36:51.735170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:36:51.735223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 00:36:51.735239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 00:36:51.735253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 00:36:51.736491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 116 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2025-02-27 00:37:03.675178: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25MiB (rounded to 2359296) Current allocation summary follows. 2025-02-27 00:37:03.684432: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-02-27 00:37:03.684499: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 60, Chunks in use: 60. 15.0KiB allocated for chunks. 15.0KiB in use in bin. 8.8KiB client-requested in use in bin. 2025-02-27 00:37:03.684526: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-02-27 00:37:03.684550: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 246, Chunks in use: 246. 246.2KiB allocated for chunks. 246.2KiB in use in bin. 246.0KiB client-requested in use in bin. 2025-02-27 00:37:03.684572: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin. 2025-02-27 00:37:03.684594: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 115, Chunks in use: 115. 465.5KiB allocated for chunks. 465.5KiB in use in bin. 460.0KiB client-requested in use in bin. 2025-02-27 00:37:03.684616: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 1, Chunks in use: 0. 14.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-02-27 00:37:03.684638: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2025-02-27 00:37:03.684658: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-02-27 00:37:03.684683: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-02-27 00:37:03.684704: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-02-27 00:37:03.684724: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 7, Chunks in use: 7. 1.91MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2025-02-27 00:37:03.684745: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2025-02-27 00:37:03.684783: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 46, Chunks in use: 44. 51.88MiB allocated for chunks. 49.62MiB in use in bin. 44.00MiB client-requested in use in bin. 2025-02-27 00:37:03.684807: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 23, Chunks in use: 23. 52.00MiB allocated for chunks. 52.00MiB in use in bin. 51.50MiB client-requested in use in bin. 2025-02-27 00:37:03.684827: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 1, Chunks in use: 1. 4.38MiB allocated for chunks. 4.38MiB in use in bin. 2.25MiB client-requested in use in bin. 2025-02-27 00:37:03.684849: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-02-27 00:37:03.684867: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-02-27 00:37:03.684885: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-02-27 00:37:03.684905: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-02-27 00:37:03.684923: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-02-27 00:37:03.684943: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-02-27 00:37:03.684963: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.25MiB was 2.00MiB, Chunk State: 2025-02-27 00:37:03.684979: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 56492032 2025-02-27 00:37:03.685004: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ec000000 of size 2359296 next 370 2025-02-27 00:37:03.685020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ec240000 of size 1048576 next 394 2025-02-27 00:37:03.685035: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7ff2ec340000 of size 1310720 next 388 2025-02-27 00:37:03.685052: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ec480000 of size 2359296 next 387 2025-02-27 00:37:03.685069: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ec6c0000 of size 1048576 next 412 2025-02-27 00:37:03.685084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ec7c0000 of size 1310720 next 406 2025-02-27 00:37:03.685102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ec900000 of size 2359296 next 405 2025-02-27 00:37:03.685119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ecb40000 of size 1048576 next 430 2025-02-27 00:37:03.685134: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ecc40000 of size 1310720 next 424 2025-02-27 00:37:03.685149: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ecd80000 of size 2359296 next 423 2025-02-27 00:37:03.685168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ecfc0000 of size 1048576 next 447 2025-02-27 00:37:03.685183: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ed0c0000 of size 1310720 next 441 2025-02-27 00:37:03.685198: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ed200000 of size 2359296 next 440 2025-02-27 00:37:03.685215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ed440000 of size 1048576 next 465 2025-02-27 00:37:03.685254: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ed540000 of size 1310720 next 459 2025-02-27 00:37:03.685273: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ed680000 of size 2359296 next 458 2025-02-27 00:37:03.685288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ed8c0000 of size 1048576 next 483 2025-02-27 00:37:03.685303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ed9c0000 of size 1310720 next 477 2025-02-27 00:37:03.685318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2edb00000 of size 2359296 next 476 2025-02-27 00:37:03.685337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2edd40000 of size 1048576 next 501 2025-02-27 00:37:03.685352: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ede40000 of size 1310720 next 495 2025-02-27 00:37:03.685367: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2edf80000 of size 2359296 next 494 2025-02-27 00:37:03.685382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ee1c0000 of size 1048576 next 519 2025-02-27 00:37:03.685399: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ee2c0000 of size 1310720 next 513 2025-02-27 00:37:03.685414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ee400000 of size 2359296 next 512 2025-02-27 00:37:03.685429: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ee640000 of size 1048576 next 537 2025-02-27 00:37:03.685448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ee740000 of size 1310720 next 531 2025-02-27 00:37:03.685464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ee880000 of size 2359296 next 530 2025-02-27 00:37:03.685478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2eeac0000 of size 1048576 next 555 2025-02-27 00:37:03.685495: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2eebc0000 of size 1310720 next 549 2025-02-27 00:37:03.685511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2eed00000 of size 2359296 next 548 2025-02-27 00:37:03.685527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2eef40000 of size 1048576 next 572 2025-02-27 00:37:03.685542: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ef040000 of size 1310720 next 567 2025-02-27 00:37:03.685560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2ef180000 of size 4587520 next 18446744073709551615 2025-02-27 00:37:03.685575: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432 2025-02-27 00:37:03.685592: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0000000 of size 2359296 next 245 2025-02-27 00:37:03.685610: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0240000 of size 1048576 next 269 2025-02-27 00:37:03.685625: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0340000 of size 1310720 next 263 2025-02-27 00:37:03.685640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0480000 of size 2359296 next 262 2025-02-27 00:37:03.685658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f06c0000 of size 1048576 next 287 2025-02-27 00:37:03.685673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f07c0000 of size 1310720 next 281 2025-02-27 00:37:03.685688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0900000 of size 2359296 next 280 2025-02-27 00:37:03.685704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0b40000 of size 1048576 next 304 2025-02-27 00:37:03.685721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0c40000 of size 1310720 next 298 2025-02-27 00:37:03.685736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0d80000 of size 2359296 next 297 2025-02-27 00:37:03.685752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f0fc0000 of size 1048576 next 322 2025-02-27 00:37:03.685780: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f10c0000 of size 1310720 next 316 2025-02-27 00:37:03.685796: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f1200000 of size 2359296 next 315 2025-02-27 00:37:03.685814: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f1440000 of size 1048576 next 340 2025-02-27 00:37:03.685830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f1540000 of size 1310720 next 334 2025-02-27 00:37:03.685845: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f1680000 of size 2359296 next 333 2025-02-27 00:37:03.685861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f18c0000 of size 1048576 next 358 2025-02-27 00:37:03.685877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f19c0000 of size 1310720 next 352 2025-02-27 00:37:03.685892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f1b00000 of size 2359296 next 351 2025-02-27 00:37:03.685908: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f1d40000 of size 1048576 next 376 2025-02-27 00:37:03.685927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f1e40000 of size 1835008 next 18446744073709551615 2025-02-27 00:37:03.685943: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216 2025-02-27 00:37:03.685957: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f2000000 of size 2359296 next 181 2025-02-27 00:37:03.685976: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f2240000 of size 2097152 next 196 2025-02-27 00:37:03.685992: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f2440000 of size 1048576 next 216 2025-02-27 00:37:03.686007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f2540000 of size 1310720 next 209 2025-02-27 00:37:03.686023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f2680000 of size 2359296 next 208 2025-02-27 00:37:03.686040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f28c0000 of size 1048576 next 234 2025-02-27 00:37:03.686055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f29c0000 of size 1310720 next 228 2025-02-27 00:37:03.686071: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f2b00000 of size 2359296 next 227 2025-02-27 00:37:03.686090: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f2d40000 of size 2883584 next 18446744073709551615 2025-02-27 00:37:03.686104: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-02-27 00:37:03.686120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4600000 of size 147456 next 55 2025-02-27 00:37:03.686139: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4624000 of size 65536 next 78 2025-02-27 00:37:03.686155: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4634000 of size 4096 next 191 2025-02-27 00:37:03.686170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4635000 of size 4096 next 192 2025-02-27 00:37:03.686185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4636000 of size 4096 next 193 2025-02-27 00:37:03.686203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4637000 of size 256 next 194 2025-02-27 00:37:03.686218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4637100 of size 256 next 195 2025-02-27 00:37:03.686233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4637200 of size 4096 next 197 2025-02-27 00:37:03.686252: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4638200 of size 4096 next 198 2025-02-27 00:37:03.686267: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4639200 of size 4096 next 199 2025-02-27 00:37:03.686311: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463a200 of size 4096 next 200 2025-02-27 00:37:03.686328: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463b200 of size 4096 next 201 2025-02-27 00:37:03.686344: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463c200 of size 1024 next 202 2025-02-27 00:37:03.686359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463c600 of size 1024 next 204 2025-02-27 00:37:03.686376: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463ca00 of size 1024 next 205 2025-02-27 00:37:03.686392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463ce00 of size 1024 next 206 2025-02-27 00:37:03.686407: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463d200 of size 1024 next 207 2025-02-27 00:37:03.686426: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463d600 of size 1024 next 210 2025-02-27 00:37:03.686442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463da00 of size 1024 next 211 2025-02-27 00:37:03.686456: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463de00 of size 1024 next 212 2025-02-27 00:37:03.686473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463e200 of size 1024 next 213 2025-02-27 00:37:03.686489: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463e600 of size 1024 next 214 2025-02-27 00:37:03.686505: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463ea00 of size 4096 next 215 2025-02-27 00:37:03.686520: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f463fa00 of size 4096 next 217 2025-02-27 00:37:03.686537: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4640a00 of size 4096 next 218 2025-02-27 00:37:03.686553: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4641a00 of size 4096 next 219 2025-02-27 00:37:03.686567: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4642a00 of size 4096 next 220 2025-02-27 00:37:03.686585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4643a00 of size 1024 next 221 2025-02-27 00:37:03.686602: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4643e00 of size 1024 next 223 2025-02-27 00:37:03.686616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4644200 of size 1024 next 224 2025-02-27 00:37:03.686632: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4644600 of size 1024 next 225 2025-02-27 00:37:03.686649: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4644a00 of size 1024 next 226 2025-02-27 00:37:03.686665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4644e00 of size 1024 next 229 2025-02-27 00:37:03.686680: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4645200 of size 1024 next 230 2025-02-27 00:37:03.686695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4645600 of size 1024 next 231 2025-02-27 00:37:03.686712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4645a00 of size 1024 next 232 2025-02-27 00:37:03.686726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4645e00 of size 1024 next 233 2025-02-27 00:37:03.686743: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4646200 of size 7680 next 72 2025-02-27 00:37:03.686761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4648000 of size 147456 next 71 2025-02-27 00:37:03.686777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f466c000 of size 131072 next 87 2025-02-27 00:37:03.686793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f468c000 of size 4096 next 289 2025-02-27 00:37:03.686811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f468d000 of size 4096 next 290 2025-02-27 00:37:03.686826: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f468e000 of size 4096 next 291 2025-02-27 00:37:03.686853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f468f000 of size 1024 next 292 2025-02-27 00:37:03.686870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f468f400 of size 1024 next 293 2025-02-27 00:37:03.686885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f468f800 of size 1024 next 294 2025-02-27 00:37:03.686920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f468fc00 of size 1024 next 295 2025-02-27 00:37:03.686942: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4690000 of size 1024 next 296 2025-02-27 00:37:03.686957: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4690400 of size 1024 next 299 2025-02-27 00:37:03.686973: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4690800 of size 1024 next 300 2025-02-27 00:37:03.686991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4690c00 of size 1024 next 301 2025-02-27 00:37:03.687006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4691000 of size 1024 next 302 2025-02-27 00:37:03.687021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4691400 of size 1024 next 303 2025-02-27 00:37:03.687037: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4691800 of size 4096 next 305 2025-02-27 00:37:03.687053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4692800 of size 4096 next 306 2025-02-27 00:37:03.687067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4693800 of size 4096 next 307 2025-02-27 00:37:03.687085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4694800 of size 4096 next 308 2025-02-27 00:37:03.687102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4695800 of size 4096 next 309 2025-02-27 00:37:03.687117: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4696800 of size 1024 next 310 2025-02-27 00:37:03.687133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4696c00 of size 1024 next 311 2025-02-27 00:37:03.687151: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4697000 of size 1024 next 312 2025-02-27 00:37:03.687166: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4697400 of size 1024 next 313 2025-02-27 00:37:03.687181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4697800 of size 1024 next 314 2025-02-27 00:37:03.687197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4697c00 of size 1024 next 317 2025-02-27 00:37:03.687214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4698000 of size 1024 next 318 2025-02-27 00:37:03.687228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4698400 of size 1024 next 319 2025-02-27 00:37:03.687244: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4698800 of size 1024 next 320 2025-02-27 00:37:03.687262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4698c00 of size 1024 next 321 2025-02-27 00:37:03.687277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4699000 of size 4096 next 323 2025-02-27 00:37:03.687292: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469a000 of size 4096 next 324 2025-02-27 00:37:03.687306: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469b000 of size 4096 next 325 2025-02-27 00:37:03.687325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469c000 of size 4096 next 326 2025-02-27 00:37:03.687340: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469d000 of size 4096 next 327 2025-02-27 00:37:03.687355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469e000 of size 1024 next 328 2025-02-27 00:37:03.687372: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469e400 of size 1024 next 329 2025-02-27 00:37:03.687398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469e800 of size 1024 next 330 2025-02-27 00:37:03.687415: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469ec00 of size 1024 next 331 2025-02-27 00:37:03.687433: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469f000 of size 1024 next 332 2025-02-27 00:37:03.687448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469f400 of size 1024 next 335 2025-02-27 00:37:03.687463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469f800 of size 1024 next 336 2025-02-27 00:37:03.687481: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f469fc00 of size 1024 next 337 2025-02-27 00:37:03.687497: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a0000 of size 1024 next 338 2025-02-27 00:37:03.687512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a0400 of size 1024 next 339 2025-02-27 00:37:03.687527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a0800 of size 4096 next 341 2025-02-27 00:37:03.687543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a1800 of size 4096 next 342 2025-02-27 00:37:03.687558: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a2800 of size 4096 next 343 2025-02-27 00:37:03.687574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a3800 of size 4096 next 344 2025-02-27 00:37:03.687592: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a4800 of size 4096 next 345 2025-02-27 00:37:03.687607: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a5800 of size 1024 next 346 2025-02-27 00:37:03.687622: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a5c00 of size 1024 next 347 2025-02-27 00:37:03.687640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a6000 of size 1024 next 348 2025-02-27 00:37:03.687656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a6400 of size 1024 next 349 2025-02-27 00:37:03.687671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a6800 of size 1024 next 350 2025-02-27 00:37:03.687687: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a6c00 of size 1024 next 353 2025-02-27 00:37:03.687704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a7000 of size 1024 next 354 2025-02-27 00:37:03.687718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a7400 of size 1024 next 355 2025-02-27 00:37:03.687735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a7800 of size 1024 next 356 2025-02-27 00:37:03.687752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a7c00 of size 1024 next 357 2025-02-27 00:37:03.687768: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a8000 of size 4096 next 359 2025-02-27 00:37:03.687783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46a9000 of size 4096 next 360 2025-02-27 00:37:03.687801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46aa000 of size 4096 next 361 2025-02-27 00:37:03.687817: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ab000 of size 4096 next 362 2025-02-27 00:37:03.687832: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ac000 of size 4096 next 363 2025-02-27 00:37:03.687847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ad000 of size 1024 next 364 2025-02-27 00:37:03.687864: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ad400 of size 1024 next 365 2025-02-27 00:37:03.687879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ad800 of size 1024 next 366 2025-02-27 00:37:03.687895: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46adc00 of size 1024 next 367 2025-02-27 00:37:03.687924: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ae000 of size 1024 next 368 2025-02-27 00:37:03.687940: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ae400 of size 1024 next 371 2025-02-27 00:37:03.687958: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ae800 of size 1024 next 372 2025-02-27 00:37:03.687974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46aec00 of size 1024 next 373 2025-02-27 00:37:03.687989: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46af000 of size 1024 next 374 2025-02-27 00:37:03.688005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46af400 of size 1024 next 375 2025-02-27 00:37:03.688022: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46af800 of size 4096 next 377 2025-02-27 00:37:03.688036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b0800 of size 4096 next 378 2025-02-27 00:37:03.688053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b1800 of size 4096 next 379 2025-02-27 00:37:03.688070: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b2800 of size 4096 next 380 2025-02-27 00:37:03.688086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b3800 of size 4096 next 381 2025-02-27 00:37:03.688100: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b4800 of size 1024 next 382 2025-02-27 00:37:03.688118: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b4c00 of size 1024 next 383 2025-02-27 00:37:03.688134: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b5000 of size 1024 next 384 2025-02-27 00:37:03.688150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b5400 of size 1024 next 385 2025-02-27 00:37:03.688165: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b5800 of size 1024 next 386 2025-02-27 00:37:03.688182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b5c00 of size 1024 next 389 2025-02-27 00:37:03.688196: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b6000 of size 1024 next 390 2025-02-27 00:37:03.688214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b6400 of size 1024 next 391 2025-02-27 00:37:03.688231: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b6800 of size 1024 next 392 2025-02-27 00:37:03.688246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b6c00 of size 1024 next 393 2025-02-27 00:37:03.688261: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b7000 of size 4096 next 395 2025-02-27 00:37:03.688279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b8000 of size 4096 next 396 2025-02-27 00:37:03.688294: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46b9000 of size 4096 next 397 2025-02-27 00:37:03.688309: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ba000 of size 4096 next 398 2025-02-27 00:37:03.688326: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bb000 of size 4096 next 399 2025-02-27 00:37:03.688342: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bc000 of size 1024 next 400 2025-02-27 00:37:03.688357: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bc400 of size 1024 next 401 2025-02-27 00:37:03.688376: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bc800 of size 1024 next 402 2025-02-27 00:37:03.688392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bcc00 of size 1024 next 403 2025-02-27 00:37:03.688407: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bd000 of size 1024 next 404 2025-02-27 00:37:03.688423: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bd400 of size 1024 next 407 2025-02-27 00:37:03.688448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bd800 of size 1024 next 408 2025-02-27 00:37:03.688464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bdc00 of size 1024 next 409 2025-02-27 00:37:03.688480: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46be000 of size 1024 next 410 2025-02-27 00:37:03.688496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46be400 of size 1024 next 411 2025-02-27 00:37:03.688510: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46be800 of size 4096 next 413 2025-02-27 00:37:03.688527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46bf800 of size 4096 next 414 2025-02-27 00:37:03.688544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c0800 of size 4096 next 415 2025-02-27 00:37:03.688560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c1800 of size 4096 next 416 2025-02-27 00:37:03.688575: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c2800 of size 4096 next 417 2025-02-27 00:37:03.688593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c3800 of size 1024 next 418 2025-02-27 00:37:03.688609: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c3c00 of size 1024 next 419 2025-02-27 00:37:03.688624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c4000 of size 1024 next 420 2025-02-27 00:37:03.688640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c4400 of size 1024 next 421 2025-02-27 00:37:03.688656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c4800 of size 1024 next 422 2025-02-27 00:37:03.688671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c4c00 of size 1024 next 425 2025-02-27 00:37:03.688690: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c5000 of size 1024 next 426 2025-02-27 00:37:03.688705: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c5400 of size 1024 next 427 2025-02-27 00:37:03.688720: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c5800 of size 1024 next 428 2025-02-27 00:37:03.688737: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c5c00 of size 1024 next 429 2025-02-27 00:37:03.688753: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c6000 of size 4096 next 431 2025-02-27 00:37:03.688769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c7000 of size 4096 next 432 2025-02-27 00:37:03.688784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c8000 of size 4096 next 433 2025-02-27 00:37:03.688800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46c9000 of size 4096 next 434 2025-02-27 00:37:03.688816: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46ca000 of size 4096 next 435 2025-02-27 00:37:03.688830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46cb000 of size 1024 next 436 2025-02-27 00:37:03.688848: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46cb400 of size 1024 next 437 2025-02-27 00:37:03.688864: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46cb800 of size 1024 next 438 2025-02-27 00:37:03.688879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46cbc00 of size 1024 next 103 2025-02-27 00:37:03.688897: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f46cc000 of size 262144 next 102 2025-02-27 00:37:03.688915: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7ff2f470c000 of size 999424 next 18446744073709551615 2025-02-27 00:37:03.688931: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-02-27 00:37:03.688947: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4800000 of size 589824 next 96 2025-02-27 00:37:03.688975: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4890000 of size 262144 next 117 2025-02-27 00:37:03.688991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d0000 of size 1024 next 439 2025-02-27 00:37:03.689010: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d0400 of size 1024 next 442 2025-02-27 00:37:03.689027: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d0800 of size 1024 next 443 2025-02-27 00:37:03.689041: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d0c00 of size 1024 next 444 2025-02-27 00:37:03.689058: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d1000 of size 1024 next 445 2025-02-27 00:37:03.689074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d1400 of size 1024 next 446 2025-02-27 00:37:03.689090: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d1800 of size 4096 next 448 2025-02-27 00:37:03.689105: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d2800 of size 4096 next 449 2025-02-27 00:37:03.689121: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d3800 of size 4096 next 450 2025-02-27 00:37:03.689137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d4800 of size 4096 next 451 2025-02-27 00:37:03.689152: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d5800 of size 4096 next 452 2025-02-27 00:37:03.689170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d6800 of size 1024 next 453 2025-02-27 00:37:03.689186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d6c00 of size 1024 next 454 2025-02-27 00:37:03.689201: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d7000 of size 1024 next 455 2025-02-27 00:37:03.689217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d7400 of size 1024 next 456 2025-02-27 00:37:03.689234: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d7800 of size 1024 next 457 2025-02-27 00:37:03.689250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d7c00 of size 1024 next 460 2025-02-27 00:37:03.689265: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d8000 of size 1024 next 461 2025-02-27 00:37:03.689281: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d8400 of size 1024 next 462 2025-02-27 00:37:03.689297: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d8800 of size 1024 next 463 2025-02-27 00:37:03.689311: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d8c00 of size 1024 next 464 2025-02-27 00:37:03.689325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48d9000 of size 4096 next 466 2025-02-27 00:37:03.689343: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48da000 of size 4096 next 467 2025-02-27 00:37:03.689359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48db000 of size 4096 next 468 2025-02-27 00:37:03.689374: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48dc000 of size 4096 next 469 2025-02-27 00:37:03.689389: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48dd000 of size 4096 next 470 2025-02-27 00:37:03.689408: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48de000 of size 1024 next 471 2025-02-27 00:37:03.689423: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48de400 of size 1024 next 472 2025-02-27 00:37:03.689438: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48de800 of size 1024 next 473 2025-02-27 00:37:03.689454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48dec00 of size 1024 next 474 2025-02-27 00:37:03.689470: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48df000 of size 1024 next 475 2025-02-27 00:37:03.689485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48df400 of size 1024 next 478 2025-02-27 00:37:03.689514: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48df800 of size 1024 next 479 2025-02-27 00:37:03.689531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48dfc00 of size 1024 next 480 2025-02-27 00:37:03.689546: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e0000 of size 1024 next 481 2025-02-27 00:37:03.689564: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e0400 of size 1024 next 482 2025-02-27 00:37:03.689580: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e0800 of size 4096 next 484 2025-02-27 00:37:03.689595: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e1800 of size 4096 next 485 2025-02-27 00:37:03.689610: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e2800 of size 4096 next 486 2025-02-27 00:37:03.689627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e3800 of size 4096 next 487 2025-02-27 00:37:03.689642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e4800 of size 4096 next 488 2025-02-27 00:37:03.689657: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e5800 of size 1024 next 489 2025-02-27 00:37:03.689675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e5c00 of size 1024 next 490 2025-02-27 00:37:03.689691: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e6000 of size 1024 next 491 2025-02-27 00:37:03.689705: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e6400 of size 1024 next 492 2025-02-27 00:37:03.689724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e6800 of size 1024 next 493 2025-02-27 00:37:03.689739: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e6c00 of size 1024 next 496 2025-02-27 00:37:03.689754: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e7000 of size 1024 next 497 2025-02-27 00:37:03.689770: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e7400 of size 1024 next 498 2025-02-27 00:37:03.689787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e7800 of size 1024 next 499 2025-02-27 00:37:03.689801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e7c00 of size 1024 next 500 2025-02-27 00:37:03.689818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e8000 of size 4096 next 502 2025-02-27 00:37:03.689836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48e9000 of size 4096 next 503 2025-02-27 00:37:03.689851: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ea000 of size 4096 next 504 2025-02-27 00:37:03.689866: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48eb000 of size 4096 next 505 2025-02-27 00:37:03.689884: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ec000 of size 4096 next 506 2025-02-27 00:37:03.689900: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ed000 of size 1024 next 507 2025-02-27 00:37:03.689940: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ed400 of size 1024 next 508 2025-02-27 00:37:03.689957: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ed800 of size 1024 next 509 2025-02-27 00:37:03.689974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48edc00 of size 1024 next 510 2025-02-27 00:37:03.689991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ee000 of size 1024 next 511 2025-02-27 00:37:03.690006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ee400 of size 1024 next 514 2025-02-27 00:37:03.690021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ee800 of size 1024 next 515 2025-02-27 00:37:03.690040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48eec00 of size 1024 next 516 2025-02-27 00:37:03.690064: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ef000 of size 1024 next 517 2025-02-27 00:37:03.690080: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ef400 of size 1024 next 518 2025-02-27 00:37:03.690097: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ef800 of size 4096 next 520 2025-02-27 00:37:03.690112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f0800 of size 4096 next 521 2025-02-27 00:37:03.690128: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f1800 of size 4096 next 522 2025-02-27 00:37:03.690146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f2800 of size 4096 next 523 2025-02-27 00:37:03.690161: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f3800 of size 4096 next 524 2025-02-27 00:37:03.690176: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f4800 of size 1024 next 525 2025-02-27 00:37:03.690194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f4c00 of size 1024 next 526 2025-02-27 00:37:03.690209: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f5000 of size 1024 next 527 2025-02-27 00:37:03.690224: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f5400 of size 1024 next 528 2025-02-27 00:37:03.690240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f5800 of size 1024 next 529 2025-02-27 00:37:03.690257: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f5c00 of size 1024 next 532 2025-02-27 00:37:03.690271: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f6000 of size 1024 next 533 2025-02-27 00:37:03.690288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f6400 of size 1024 next 534 2025-02-27 00:37:03.690305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f6800 of size 1024 next 535 2025-02-27 00:37:03.690320: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f6c00 of size 1024 next 536 2025-02-27 00:37:03.690334: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f7000 of size 4096 next 538 2025-02-27 00:37:03.690353: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f8000 of size 4096 next 539 2025-02-27 00:37:03.690368: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48f9000 of size 4096 next 540 2025-02-27 00:37:03.690383: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fa000 of size 4096 next 541 2025-02-27 00:37:03.690399: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fb000 of size 4096 next 542 2025-02-27 00:37:03.690415: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fc000 of size 1024 next 543 2025-02-27 00:37:03.690430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fc400 of size 1024 next 544 2025-02-27 00:37:03.690446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fc800 of size 1024 next 545 2025-02-27 00:37:03.690463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fcc00 of size 1024 next 546 2025-02-27 00:37:03.690479: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fd000 of size 1024 next 547 2025-02-27 00:37:03.690493: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fd400 of size 1024 next 550 2025-02-27 00:37:03.690512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fd800 of size 1024 next 551 2025-02-27 00:37:03.690527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fdc00 of size 1024 next 552 2025-02-27 00:37:03.690542: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fe000 of size 1024 next 553 2025-02-27 00:37:03.690558: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fe400 of size 1024 next 554 2025-02-27 00:37:03.690584: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48fe800 of size 4096 next 556 2025-02-27 00:37:03.690601: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f48ff800 of size 4096 next 557 2025-02-27 00:37:03.690619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4900800 of size 4096 next 558 2025-02-27 00:37:03.690634: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4901800 of size 4096 next 559 2025-02-27 00:37:03.690649: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4902800 of size 4096 next 560 2025-02-27 00:37:03.690667: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4903800 of size 1024 next 561 2025-02-27 00:37:03.690683: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4903c00 of size 1024 next 562 2025-02-27 00:37:03.690698: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4904000 of size 1024 next 563 2025-02-27 00:37:03.690714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4904400 of size 1024 next 564 2025-02-27 00:37:03.690731: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4904800 of size 1024 next 565 2025-02-27 00:37:03.690746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4904c00 of size 1024 next 566 2025-02-27 00:37:03.690759: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4905000 of size 1024 next 568 2025-02-27 00:37:03.690771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4905400 of size 1024 next 569 2025-02-27 00:37:03.690783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4905800 of size 1024 next 570 2025-02-27 00:37:03.690794: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4905c00 of size 1024 next 571 2025-02-27 00:37:03.690806: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4906000 of size 4096 next 573 2025-02-27 00:37:03.690818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4907000 of size 4096 next 574 2025-02-27 00:37:03.690829: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4908000 of size 4096 next 575 2025-02-27 00:37:03.690841: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4909000 of size 4096 next 576 2025-02-27 00:37:03.690853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f490a000 of size 4096 next 577 2025-02-27 00:37:03.690864: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f490b000 of size 1024 next 578 2025-02-27 00:37:03.690876: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f490b400 of size 1024 next 579 2025-02-27 00:37:03.690888: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f490b800 of size 1024 next 580 2025-02-27 00:37:03.690939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f490bc00 of size 1024 next 581 2025-02-27 00:37:03.690956: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f490c000 of size 1024 next 582 2025-02-27 00:37:03.690971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f490c400 of size 256 next 583 2025-02-27 00:37:03.690986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7ff2f490c500 of size 15104 next 112 2025-02-27 00:37:03.691001: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4910000 of size 524288 next 111 2025-02-27 00:37:03.691014: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4990000 of size 262144 next 135 2025-02-27 00:37:03.691026: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f49d0000 of size 327680 next 124 2025-02-27 00:37:03.691038: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4a20000 of size 589824 next 123 2025-02-27 00:37:03.691050: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4ab0000 of size 262144 next 152 2025-02-27 00:37:03.691072: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4af0000 of size 327680 next 142 2025-02-27 00:37:03.691085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4b40000 of size 786432 next 18446744073709551615 2025-02-27 00:37:03.691096: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-02-27 00:37:03.691109: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7ff2f4c00000 of size 589824 next 160 2025-02-27 00:37:03.691121: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4c90000 of size 589824 next 158 2025-02-27 00:37:03.691133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4d20000 of size 524288 next 171 2025-02-27 00:37:03.691145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7ff2f4da0000 of size 1048576 next 222 2025-02-27 00:37:03.691157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4ea0000 of size 1048576 next 190 2025-02-27 00:37:03.691169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f4fa0000 of size 1048576 next 189 2025-02-27 00:37:03.691180: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f50a0000 of size 1048576 next 203 2025-02-27 00:37:03.691192: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f51a0000 of size 1048576 next 251 2025-02-27 00:37:03.691204: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff2f52a0000 of size 1441792 next 18446744073709551615 2025-02-27 00:37:03.691216: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-02-27 00:37:03.691228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00000 of size 1280 next 1 2025-02-27 00:37:03.691240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00500 of size 256 next 5 2025-02-27 00:37:03.691252: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00600 of size 256 next 8 2025-02-27 00:37:03.691264: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00700 of size 256 next 9 2025-02-27 00:37:03.691275: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00800 of size 256 next 10 2025-02-27 00:37:03.691287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00900 of size 256 next 11 2025-02-27 00:37:03.691299: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00a00 of size 256 next 12 2025-02-27 00:37:03.691310: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00b00 of size 256 next 13 2025-02-27 00:37:03.691322: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00c00 of size 256 next 17 2025-02-27 00:37:03.691334: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00d00 of size 256 next 19 2025-02-27 00:37:03.691345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00e00 of size 256 next 20 2025-02-27 00:37:03.691357: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae00f00 of size 256 next 21 2025-02-27 00:37:03.691369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01000 of size 256 next 22 2025-02-27 00:37:03.691380: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01100 of size 256 next 24 2025-02-27 00:37:03.691392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01200 of size 256 next 25 2025-02-27 00:37:03.691404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01300 of size 256 next 23 2025-02-27 00:37:03.691416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01400 of size 256 next 28 2025-02-27 00:37:03.691427: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01500 of size 256 next 29 2025-02-27 00:37:03.691439: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01600 of size 256 next 30 2025-02-27 00:37:03.691451: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01700 of size 256 next 31 2025-02-27 00:37:03.691471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01800 of size 256 next 33 2025-02-27 00:37:03.691483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01900 of size 256 next 34 2025-02-27 00:37:03.691495: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01a00 of size 1024 next 32 2025-02-27 00:37:03.691507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae01e00 of size 1024 next 37 2025-02-27 00:37:03.691518: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae02200 of size 1024 next 38 2025-02-27 00:37:03.691530: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae02600 of size 1024 next 39 2025-02-27 00:37:03.691542: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae02a00 of size 1024 next 40 2025-02-27 00:37:03.691553: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae02e00 of size 1024 next 41 2025-02-27 00:37:03.691565: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae03200 of size 1024 next 43 2025-02-27 00:37:03.691577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae03600 of size 1024 next 44 2025-02-27 00:37:03.691588: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae03a00 of size 1024 next 45 2025-02-27 00:37:03.691600: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae03e00 of size 1024 next 46 2025-02-27 00:37:03.691612: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04200 of size 256 next 48 2025-02-27 00:37:03.691624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04300 of size 256 next 49 2025-02-27 00:37:03.691636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04400 of size 256 next 50 2025-02-27 00:37:03.691647: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04500 of size 256 next 51 2025-02-27 00:37:03.691659: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04600 of size 256 next 52 2025-02-27 00:37:03.691671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04700 of size 256 next 53 2025-02-27 00:37:03.691682: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04800 of size 256 next 56 2025-02-27 00:37:03.691694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04900 of size 256 next 57 2025-02-27 00:37:03.691706: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04a00 of size 256 next 58 2025-02-27 00:37:03.691717: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04b00 of size 256 next 14 2025-02-27 00:37:03.691729: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04c00 of size 256 next 15 2025-02-27 00:37:03.691741: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04d00 of size 256 next 16 2025-02-27 00:37:03.691752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae04e00 of size 1024 next 60 2025-02-27 00:37:03.691764: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae05200 of size 1024 next 61 2025-02-27 00:37:03.691776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae05600 of size 1024 next 62 2025-02-27 00:37:03.691787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae05a00 of size 1024 next 63 2025-02-27 00:37:03.691799: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae05e00 of size 1024 next 64 2025-02-27 00:37:03.691811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06200 of size 256 next 66 2025-02-27 00:37:03.691823: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06300 of size 256 next 67 2025-02-27 00:37:03.691834: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06400 of size 256 next 68 2025-02-27 00:37:03.691858: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06500 of size 256 next 69 2025-02-27 00:37:03.691870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06600 of size 256 next 70 2025-02-27 00:37:03.691882: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06700 of size 256 next 73 2025-02-27 00:37:03.691894: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06800 of size 256 next 74 2025-02-27 00:37:03.691905: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06900 of size 256 next 75 2025-02-27 00:37:03.691917: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06a00 of size 256 next 76 2025-02-27 00:37:03.691927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06b00 of size 256 next 77 2025-02-27 00:37:03.691937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae06c00 of size 1024 next 79 2025-02-27 00:37:03.691945: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae07000 of size 1024 next 80 2025-02-27 00:37:03.691953: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae07400 of size 1024 next 81 2025-02-27 00:37:03.691962: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae07800 of size 1024 next 82 2025-02-27 00:37:03.691970: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae07c00 of size 1024 next 83 2025-02-27 00:37:03.691979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08000 of size 256 next 85 2025-02-27 00:37:03.691987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08100 of size 256 next 86 2025-02-27 00:37:03.691996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08200 of size 512 next 84 2025-02-27 00:37:03.692004: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08400 of size 512 next 88 2025-02-27 00:37:03.692012: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08600 of size 512 next 89 2025-02-27 00:37:03.692021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08800 of size 512 next 90 2025-02-27 00:37:03.692029: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08a00 of size 512 next 91 2025-02-27 00:37:03.692037: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08c00 of size 256 next 93 2025-02-27 00:37:03.692046: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08d00 of size 256 next 94 2025-02-27 00:37:03.692054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae08e00 of size 512 next 92 2025-02-27 00:37:03.692063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae09000 of size 512 next 97 2025-02-27 00:37:03.692071: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae09200 of size 512 next 98 2025-02-27 00:37:03.692079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae09400 of size 512 next 99 2025-02-27 00:37:03.692088: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae09600 of size 512 next 2 2025-02-27 00:37:03.692096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae09800 of size 256 next 3 2025-02-27 00:37:03.692104: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae09900 of size 256 next 4 2025-02-27 00:37:03.692113: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae09a00 of size 16384 next 18 2025-02-27 00:37:03.692121: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae0da00 of size 256 next 100 2025-02-27 00:37:03.692130: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae0db00 of size 256 next 101 2025-02-27 00:37:03.692138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae0dc00 of size 2048 next 104 2025-02-27 00:37:03.692147: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae0e400 of size 2048 next 105 2025-02-27 00:37:03.692161: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae0ec00 of size 2048 next 106 2025-02-27 00:37:03.692170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae0f400 of size 2048 next 107 2025-02-27 00:37:03.692178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae0fc00 of size 2048 next 108 2025-02-27 00:37:03.692186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae10400 of size 256 next 109 2025-02-27 00:37:03.692195: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae10500 of size 256 next 110 2025-02-27 00:37:03.692203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae10600 of size 2048 next 113 2025-02-27 00:37:03.692212: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae10e00 of size 2048 next 114 2025-02-27 00:37:03.692220: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae11600 of size 2048 next 115 2025-02-27 00:37:03.692228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae11e00 of size 2048 next 116 2025-02-27 00:37:03.692237: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae12600 of size 2048 next 6 2025-02-27 00:37:03.692245: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae12e00 of size 37632 next 7 2025-02-27 00:37:03.692254: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1c100 of size 512 next 118 2025-02-27 00:37:03.692262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1c300 of size 512 next 119 2025-02-27 00:37:03.692270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1c500 of size 512 next 120 2025-02-27 00:37:03.692279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1c700 of size 512 next 121 2025-02-27 00:37:03.692287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1c900 of size 512 next 122 2025-02-27 00:37:03.692295: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1cb00 of size 512 next 125 2025-02-27 00:37:03.692304: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1cd00 of size 512 next 126 2025-02-27 00:37:03.692312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1cf00 of size 512 next 127 2025-02-27 00:37:03.692321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1d100 of size 512 next 128 2025-02-27 00:37:03.692329: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1d300 of size 512 next 129 2025-02-27 00:37:03.692337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1d500 of size 2048 next 130 2025-02-27 00:37:03.692346: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1dd00 of size 2048 next 131 2025-02-27 00:37:03.692354: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1e500 of size 2048 next 132 2025-02-27 00:37:03.692362: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1ed00 of size 2048 next 133 2025-02-27 00:37:03.692371: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1f500 of size 2048 next 134 2025-02-27 00:37:03.692379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1fd00 of size 512 next 136 2025-02-27 00:37:03.692387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae1ff00 of size 512 next 137 2025-02-27 00:37:03.692396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae20100 of size 512 next 138 2025-02-27 00:37:03.692404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae20300 of size 512 next 139 2025-02-27 00:37:03.692413: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae20500 of size 512 next 140 2025-02-27 00:37:03.692421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae20700 of size 512 next 141 2025-02-27 00:37:03.692429: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae20900 of size 512 next 143 2025-02-27 00:37:03.692443: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae20b00 of size 512 next 144 2025-02-27 00:37:03.692452: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae20d00 of size 512 next 145 2025-02-27 00:37:03.692460: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae20f00 of size 512 next 146 2025-02-27 00:37:03.692469: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae21100 of size 2048 next 147 2025-02-27 00:37:03.692477: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae21900 of size 2048 next 148 2025-02-27 00:37:03.692485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae22100 of size 2048 next 149 2025-02-27 00:37:03.692494: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae22900 of size 2048 next 150 2025-02-27 00:37:03.692502: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae23100 of size 2048 next 151 2025-02-27 00:37:03.692511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae23900 of size 512 next 153 2025-02-27 00:37:03.692519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae23b00 of size 512 next 154 2025-02-27 00:37:03.692527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae23d00 of size 512 next 155 2025-02-27 00:37:03.692536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae23f00 of size 512 next 156 2025-02-27 00:37:03.692544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae24100 of size 512 next 157 2025-02-27 00:37:03.692552: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae24300 of size 512 next 161 2025-02-27 00:37:03.692561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae24500 of size 512 next 162 2025-02-27 00:37:03.692569: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae24700 of size 512 next 163 2025-02-27 00:37:03.692578: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae24900 of size 512 next 164 2025-02-27 00:37:03.692586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae24b00 of size 512 next 165 2025-02-27 00:37:03.692594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae24d00 of size 2048 next 166 2025-02-27 00:37:03.692603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae25500 of size 2048 next 167 2025-02-27 00:37:03.692611: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae25d00 of size 2048 next 168 2025-02-27 00:37:03.692619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae26500 of size 2048 next 169 2025-02-27 00:37:03.692628: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae26d00 of size 2048 next 170 2025-02-27 00:37:03.692636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae27500 of size 1024 next 172 2025-02-27 00:37:03.692644: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae27900 of size 1024 next 173 2025-02-27 00:37:03.692653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae27d00 of size 1024 next 174 2025-02-27 00:37:03.692662: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae28100 of size 1024 next 175 2025-02-27 00:37:03.692670: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae28500 of size 1024 next 176 2025-02-27 00:37:03.692678: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae28900 of size 256 next 178 2025-02-27 00:37:03.692687: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae28a00 of size 256 next 179 2025-02-27 00:37:03.692695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae28b00 of size 1024 next 177 2025-02-27 00:37:03.692704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae28f00 of size 1024 next 182 2025-02-27 00:37:03.692718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae29300 of size 1024 next 183 2025-02-27 00:37:03.692726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae29700 of size 1024 next 184 2025-02-27 00:37:03.692735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae29b00 of size 1024 next 185 2025-02-27 00:37:03.692743: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae29f00 of size 256 next 187 2025-02-27 00:37:03.692752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae2a000 of size 256 next 188 2025-02-27 00:37:03.692760: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae2a100 of size 4096 next 186 2025-02-27 00:37:03.692768: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae2b100 of size 4096 next 42 2025-02-27 00:37:03.692777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae2c100 of size 65536 next 36 2025-02-27 00:37:03.692785: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae3c100 of size 65536 next 35 2025-02-27 00:37:03.692793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae4c100 of size 4096 next 235 2025-02-27 00:37:03.692802: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae4d100 of size 4096 next 236 2025-02-27 00:37:03.692810: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae4e100 of size 4096 next 237 2025-02-27 00:37:03.692819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae4f100 of size 4096 next 238 2025-02-27 00:37:03.692827: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae50100 of size 1024 next 239 2025-02-27 00:37:03.692835: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae50500 of size 1024 next 240 2025-02-27 00:37:03.692844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae50900 of size 1024 next 241 2025-02-27 00:37:03.692852: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae50d00 of size 1024 next 242 2025-02-27 00:37:03.692860: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae51100 of size 1024 next 243 2025-02-27 00:37:03.692869: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae51500 of size 1024 next 246 2025-02-27 00:37:03.692877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae51900 of size 1024 next 247 2025-02-27 00:37:03.692886: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae51d00 of size 1024 next 248 2025-02-27 00:37:03.692894: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae52100 of size 1024 next 249 2025-02-27 00:37:03.692902: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae52500 of size 1024 next 250 2025-02-27 00:37:03.692911: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae52900 of size 4096 next 252 2025-02-27 00:37:03.692919: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae53900 of size 4096 next 253 2025-02-27 00:37:03.692927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae54900 of size 4096 next 254 2025-02-27 00:37:03.692936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae55900 of size 4096 next 255 2025-02-27 00:37:03.692944: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae56900 of size 4096 next 256 2025-02-27 00:37:03.692953: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae57900 of size 1024 next 257 2025-02-27 00:37:03.692961: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae57d00 of size 1024 next 258 2025-02-27 00:37:03.692969: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae58100 of size 1024 next 259 2025-02-27 00:37:03.692978: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae58500 of size 1024 next 260 2025-02-27 00:37:03.692992: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae58900 of size 1024 next 261 2025-02-27 00:37:03.693000: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae58d00 of size 1024 next 264 2025-02-27 00:37:03.693009: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae59100 of size 1024 next 265 2025-02-27 00:37:03.693017: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae59500 of size 1024 next 266 2025-02-27 00:37:03.693026: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae59900 of size 1024 next 267 2025-02-27 00:37:03.693034: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae59d00 of size 1024 next 268 2025-02-27 00:37:03.693042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5a100 of size 4096 next 270 2025-02-27 00:37:03.693051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5b100 of size 4096 next 271 2025-02-27 00:37:03.693059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5c100 of size 4096 next 272 2025-02-27 00:37:03.693067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5d100 of size 4096 next 273 2025-02-27 00:37:03.693076: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5e100 of size 4096 next 274 2025-02-27 00:37:03.693084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5f100 of size 1024 next 275 2025-02-27 00:37:03.693093: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5f500 of size 1024 next 276 2025-02-27 00:37:03.693101: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5f900 of size 1024 next 277 2025-02-27 00:37:03.693109: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae5fd00 of size 1024 next 278 2025-02-27 00:37:03.693118: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae60100 of size 1024 next 279 2025-02-27 00:37:03.693126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae60500 of size 1024 next 282 2025-02-27 00:37:03.693135: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae60900 of size 1024 next 283 2025-02-27 00:37:03.693143: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae60d00 of size 1024 next 284 2025-02-27 00:37:03.693151: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae61100 of size 1024 next 285 2025-02-27 00:37:03.693160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae61500 of size 1024 next 286 2025-02-27 00:37:03.693168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae61900 of size 4096 next 288 2025-02-27 00:37:03.693177: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae62900 of size 6144 next 27 2025-02-27 00:37:03.693185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae64100 of size 147456 next 26 2025-02-27 00:37:03.693194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae88100 of size 65536 next 47 2025-02-27 00:37:03.693202: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42ae98100 of size 65536 next 59 2025-02-27 00:37:03.693210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42aea8100 of size 65536 next 65 2025-02-27 00:37:03.693219: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7ff42aeb8100 of size 294656 next 18446744073709551615 2025-02-27 00:37:03.693227: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-02-27 00:37:03.693243: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 60 Chunks of size 256 totalling 15.0KiB 2025-02-27 00:37:03.693253: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-02-27 00:37:03.693263: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 245 Chunks of size 1024 totalling 245.0KiB 2025-02-27 00:37:03.693279: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-02-27 00:37:03.693288: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB 2025-02-27 00:37:03.693298: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 113 Chunks of size 4096 totalling 452.0KiB 2025-02-27 00:37:03.693307: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 6144 totalling 6.0KiB 2025-02-27 00:37:03.693316: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2025-02-27 00:37:03.693325: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-02-27 00:37:03.693335: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-02-27 00:37:03.693344: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-02-27 00:37:03.693354: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-02-27 00:37:03.693363: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-02-27 00:37:03.693372: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2025-02-27 00:37:03.693382: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-02-27 00:37:03.693391: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-02-27 00:37:03.693400: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2025-02-27 00:37:03.693409: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-02-27 00:37:03.693418: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-02-27 00:37:03.693428: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 24 Chunks of size 1048576 totalling 24.00MiB 2025-02-27 00:37:03.693437: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 18 Chunks of size 1310720 totalling 22.50MiB 2025-02-27 00:37:03.693446: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB 2025-02-27 00:37:03.693455: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB 2025-02-27 00:37:03.693464: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2097152 totalling 2.00MiB 2025-02-27 00:37:03.693473: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 21 Chunks of size 2359296 totalling 47.25MiB 2025-02-27 00:37:03.693483: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2883584 totalling 2.75MiB 2025-02-27 00:37:03.693492: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 4587520 totalling 4.38MiB 2025-02-27 00:37:03.693501: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 113.09MiB 2025-02-27 00:37:03.693510: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 122552320 memory_limit_: 122552320 available bytes: 0 curr_region_allocation_bytes_: 134217728 2025-02-27 00:37:03.693523: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 122552320 InUse: 118588672 MaxInUse: 118588672 NumAllocs: 1959 MaxAllocSize: 4587520 2025-02-27 00:37:03.693547: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *********************************************x****************************************************** 2025-02-27 00:37:03.693595: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Mul] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 ERROR : 'int' object is not subscriptable reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: "'int' object is not subscriptable" /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_3bb8fc9eb89f6e73213a93d2f1429765ec1e113a SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_3bb8fc9eb89f6e73213a93d2f1429765ec1e113a','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/February/27022025/python_test3//data_2/data_log/job/2025/February/27022025/python_test3/log-python3----short_python3--v--marlene-00:35:01.txt','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.24843144416809082 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Thu Feb 27 00:37:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740613024_1084700_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1740613024_1084700_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 ERROR in datou_step_exec, will save and exit ! CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2430, in datou_step_exec return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 367, in datou_step_sam sam.to(device=device) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to return self._apply(convert) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 602, in _apply param_applied = fn(param) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', "[>, , , , , 'CUDA error: out of memory\\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.16429853439331055 save_final ERROR in last step sam, CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. time spend for datou_step_exec : 4.64703369140625 time spend to save output : 0.1698615550994873 total time spend for step 0 : 4.816895246505737 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : None ERROR nb objects espect : 98 nb_objects detect : 0 ERROR sam FAILED ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.321974515914917 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Thu Feb 27 00:37:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740613029_1084700_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1740613029_1084700_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] local folder : /data/models_weight/detection_plaque_valcor_010622 /data/models_weight/detection_plaque_valcor_010622/caffemodel size_local : 349723073 size in s3 : 349723073 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:56 caffemodel already exist and didn't need to update /data/models_weight/detection_plaque_valcor_010622/test.prototxt size_local : 7163 size in s3 : 7163 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:55 test.prototxt already exist and didn't need to update prototxt : /data/models_weight/detection_plaque_valcor_010622/test.prototxt caffemodel : /data/models_weight/detection_plaque_valcor_010622/caffemodel Loaded network /data/models_weight/detection_plaque_valcor_010622/caffemodel About to compute detect_faster_rcnn : len(args) : 1 Inside frcnn step exec : nb paths : 1 image_path : temp/1740613029_1084700_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg image_size (600, 800, 3) [[[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] ... [[ 14 16 16] [ 13 15 15] [ 11 13 13] ... [198 206 205] [198 206 205] [198 206 205]] [[ 16 18 18] [ 14 16 16] [ 11 13 13] ... [206 214 213] [206 214 213] [206 214 213]] [[ 13 15 15] [ 12 14 14] [ 9 11 11] ... [210 218 217] [210 218 217] [210 218 217]]] Detection took 0.080s for 300 object proposals c : plaque list_crops.shape (72, 5) proba : 0.063821845 (374.1271, 293.9192, 430.81003, 317.80743) proba : 0.052216917 (382.17572, 297.19025, 552.35144, 344.65936) proba : 0.0122745875 (345.35706, 272.43405, 468.86035, 320.72702) We are managing local photo_id len de result frcnn : 1 After datou_step_exec type output : time spend for datou_step_exec : 3.449429988861084 time spend to save output : 0.00012922286987304688 total time spend for step 1 : 3.449559211730957 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True Inside saveFrcnn : final : True verbose : True threshold to save the result : 0.1 output flattener : [(0, 493029425, 4370, 374, 430, 293, 317, 0.063821845, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.052216917, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.0122745875, None)] Warning : no hashtag_ids to insert in the database final : True begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', '0', 0, '0', 493061979, '0', None)] time used for this insertion : 0.07673931121826172 [917754606] map_info['map_portfolio_photo'] : {} final : True mtd_id 4184 list_pids : [917754606] Looping around the photos to save general results len do output : 1 /0 before output type Managing all output in save final without adding information in the mtr_datou_result ('4184', None, None, None, None, None, None, None, None) ('4184', None, '917754606', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', None, None, None, None, None, None)] time used for this insertion : 0.10089921951293945 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {0: [[(0, 493029425, 4370, 374, 430, 293, 317, 0.063821845, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.052216917, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.0122745875, None)], 'temp/1740613029_1084700_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg']} ############################### TEST thcl ################################ TEST THCL Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=2 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=2 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 2 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=2 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 1 thcl is not linked in the step_by_step architecture ! WARNING : step 2 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (916235064) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 916235064 download finish for photo 916235064 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.26461124420166016 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:thcl Thu Feb 27 00:37:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740613033_1084700_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1740613033_1084700_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'355': 1} we are using the classfication for only one thcl 355 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (66, 66, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.006645679473876953 time to convert the images to numpy array : 0.0038771629333496094 total time to convert the images to numpy array : 0.010818958282470703 list photo_ids error: [] list photo_ids correct : [916235064] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 355 To do loadFromThcl(), then load ParamDescType : thcl355 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (355) thcls : [{'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3390 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) To loadFromThcl() : net_3390 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 2425 wait 20 seconds l 3637 free memory gpu now : 2425 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) param : , param.caffemodel : car_360_1027 None mean_file_type : mean_file_path : prototxt_file_path : model : car_360_1027 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : car_360_1027 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/car_360_1027 /data/models_weight/car_360_1027/caffemodel size_local : 542944640 size in s3 : 542944640 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 caffemodel already exist and didn't need to update /data/models_weight/car_360_1027/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy_fc.prototxt size_local : 1132 size in s3 : 1132 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy.prototxt size_local : 5654 size in s3 : 5654 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/mean.npy size_local : 1572944 size in s3 : 1572944 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:55 mean.npy already exist and didn't need to update /data/models_weight/car_360_1027/synset_words.txt size_local : 13687 size in s3 : 13687 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/car_360_1027/deploy.prototxt caffemodel_filename : /data/models_weight/car_360_1027/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2644 max_wait_temp : 1 max_wait : 0 dict_keys(['pool5', 'prob']) time used to do the prepocess of the images : 0.013761043548583984 time used to do the prediction : 0.1595745086669922 save descriptor for thcl : 355 (1, 512, 7, 7) Got the blobs of the net to insert : [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] code_as_byte_string:b'0000000000'| time to traite the descriptors : 0.05312037467956543 Testing : ['916235064'] In select_photos_meta_from_ids: SELECT photo_id, url, FROM_UNIXTIME(uploaded_at), latitude, longitude, text FROM MTRBack.photos WHERE photo_id IN (916235064) result : {916235064: {'photo_id': 916235064, 'url': 'https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/2017/10/14/6293d1bb790dc6902450e7c572b7d10b.jpg', 'latitude': 0.0, 'longitude': 0.0, 'text': None}} list_photo_exists : [916235064] storage_type for insertDescriptorsMulti : 1 To insert : 916235064 time to insert the descriptors : 0.7363183498382568 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : False verbose : True time used to find the portfolios of the photos select button_legend_list from MTRDatou.classification_theme where id = 355 SAVE THCL, output : {'916235064': [[('916235064', 'c_elysee_1027_gao__port_506302', 0.0018812087, 332, '355'), ('916235064', 'mokka_1027_gao__port_506374', 0.0011634391, 332, '355'), ('916235064', 'captur_1027_gao__port_506399', 0.0008155512, 332, '355'), ('916235064', 'sorento_1027_gao__port_506192', 0.0011770271, 332, '355'), ('916235064', 'navara_1027_gao__port_506205', 0.0025851857, 332, '355'), ('916235064', 'xc90_1027_gao__port_506350', 0.004169884, 332, '355'), ('916235064', 'saxo_1027_gao__port_506052', 0.0034804896, 332, '355'), ('916235064', 'trafic_1027_gao__port_506295', 0.007368081, 332, '355'), ('916235064', 'punto_evo_1027_gao__port_506066', 0.002188991, 332, '355'), ('916235064', '5_1027_gao__port_506117', 0.00057974545, 332, '355'), ('916235064', '250_1027_gao__port_506065', 0.004591638, 332, '355'), ('916235064', 'd_max_1027_gao__port_506125', 0.003158591, 332, '355'), ('916235064', 'panamera_1027_gao__port_506387', 0.0022505983, 332, '355'), ('916235064', 'alhambra_1027_gao__port_506381', 0.0053196056, 332, '355'), ('916235064', 'x6_1027_gao__port_506349', 0.0010999705, 332, '355'), ('916235064', 'vitara_1027_gao__port_506328', 0.0054014595, 332, '355'), ('916235064', 'fiesta_1027_gao__port_506377', 0.0039192336, 332, '355'), ('916235064', 'qashqai_1027_gao__port_506286', 0.0014785927, 332, '355'), ('916235064', '147_1027_gao__port_506124', 0.0019780223, 332, '355'), ('916235064', 'c5_1027_gao__port_506172', 0.0012439169, 332, '355'), ('916235064', 'q5_1027_gao__port_506206', 0.00150478, 332, '355'), ('916235064', 'giulia_1027_gao__port_506178', 0.0021695313, 332, '355'), ('916235064', 'karl_1027_gao__port_506371', 0.0027078276, 332, '355'), ('916235064', 'mehari_1027_gao__port_506076', 0.0047043385, 332, '355'), ('916235064', '911_1027_gao__port_506114', 0.0019419208, 332, '355'), ('916235064', '508_1027_gao__port_506329', 0.00095842284, 332, '355'), ('916235064', 'idea_1027_gao__port_506122', 0.0007697986, 332, '355'), ('916235064', 'megane_1027_gao__port_506220', 0.0019468032, 332, '355'), ('916235064', 'ghibli_1027_gao__port_506174', 0.001372461, 332, '355'), ('916235064', 'touareg_1027_gao__port_506224', 0.001620137, 332, '355'), ('916235064', 'i10_1027_gao__port_506232', 0.0013923238, 332, '355'), ('916235064', 'jumper_1027_gao__port_506234', 0.010044107, 332, '355'), ('916235064', 'classe_clk_1027_gao__port_506173', 0.0010792068, 332, '355'), ('916235064', 'kuga_1027_gao__port_506181', 0.00084462366, 332, '355'), ('916235064', 'ct_1027_gao__port_506323', 0.00125208, 332, '355'), ('916235064', 'leon_1027_gao__port_506326', 0.0025842676, 332, '355'), ('916235064', 'ds5_1027_gao__port_506376', 0.0012427535, 332, '355'), ('916235064', 'cordoba_1027_gao__port_506048', 0.0028649687, 332, '355'), ('916235064', 'classe_cla_1027_gao__port_506400', 0.0012947889, 332, '355'), ('916235064', 'jumpy_1027_gao__port_506179', 0.010338067, 332, '355'), ('916235064', 'avensis_1027_gao__port_506311', 0.0018765352, 332, '355'), ('916235064', 'juke_1027_gao__port_506325', 0.0011342535, 332, '355'), ('916235064', '4008_1027_gao__port_506402', 0.0015756896, 332, '355'), ('916235064', '190_series_1027_gao__port_506051', 0.0039807996, 332, '355'), ('916235064', 'serie_3_1027_gao__port_506294', 0.002874276, 332, '355'), ('916235064', 'q7_1027_gao__port_506318', 0.0023353854, 332, '355'), ('916235064', 'glc_1027_gao__port_506303', 0.0012106374, 332, '355'), ('916235064', 'grand_vitara_1027_gao__port_506175', 0.0011445165, 332, '355'), ('916235064', 's40_1027_gao__port_506099', 0.0022336333, 332, '355'), ('916235064', 'toledo_1027_gao__port_506061', 0.0017465596, 332, '355'), ('916235064', '5008_1027_gao__port_506337', 0.0046988716, 332, '355'), ('916235064', 'continental_1027_gao__port_506250', 0.0021913303, 332, '355'), ('916235064', 'coupe_1027_gao__port_506082', 0.0022632838, 332, '355'), ('916235064', 'iq_1027_gao__port_506166', 0.0018173679, 332, '355'), ('916235064', '407_1027_gao__port_506133', 0.00090550276, 332, '355'), ('916235064', 'touran_1027_gao__port_506308', 0.0020399909, 332, '355'), ('916235064', '300c_1027_gao__port_506078', 0.0025335269, 332, '355'), ('916235064', 'classe_gl_1027_gao__port_506340', 0.0044887504, 332, '355'), ('916235064', 'vivaro_1027_gao__port_506310', 0.003425516, 332, '355'), ('916235064', 'sl_1027_gao__port_506100', 0.003135614, 332, '355'), ('916235064', 'elise_1027_gao__port_506121', 0.0010256472, 332, '355'), ('916235064', '1007_1027_gao__port_506070', 0.0015352357, 332, '355'), ('916235064', 'i40_1027_gao__port_506218', 0.0005913617, 332, '355'), ('916235064', 'bipper_tepee_1027_gao__port_506227', 0.0040292046, 332, '355'), ('916235064', 'focus_1027_gao__port_506272', 0.0011584308, 332, '355'), ('916235064', 'primera_1027_gao__port_506147', 0.0012156733, 332, '355'), ('916235064', 'r4_1027_gao__port_506160', 0.014970732, 332, '355'), ('916235064', 'a8_1027_gao__port_506265', 0.0011319184, 332, '355'), ('916235064', 'boxer_1027_gao__port_506202', 0.01054657, 332, '355'), ('916235064', 's5_1027_gao__port_506222', 0.0011985191, 332, '355'), ('916235064', 'r21_1027_gao__port_506093', 0.004185603, 332, '355'), ('916235064', 'c3_1027_gao__port_506257', 0.0023629856, 332, '355'), ('916235064', 'santa_fe_1027_gao__port_506208', 0.0016321875, 332, '355'), ('916235064', 'm4_1027_gao__port_506344', 0.0015567197, 332, '355'), ('916235064', 'safrane_1027_gao__port_506077', 0.0013958018, 332, '355'), ('916235064', 'classe_gle_1027_gao__port_506395', 0.002197798, 332, '355'), ('916235064', '0_1027_gao__port_506094', 0.008828037, 332, '355'), ('916235064', 'ix35_1027_gao__port_506219', 0.0014612252, 332, '355'), ('916235064', 'carens_1027_gao__port_506298', 0.00088231545, 332, '355'), ('916235064', 'classe_a_1027_gao__port_506339', 0.002471064, 332, '355'), ('916235064', 'ix20_1027_gao__port_506343', 0.001009093, 332, '355'), ('916235064', 'note_1027_gao__port_506365', 0.0015961025, 332, '355'), ('916235064', 'a5_1027_gao__port_506200', 0.0015330603, 332, '355'), ('916235064', 'sx4_1027_gao__port_506348', 0.0014913596, 332, '355'), ('916235064', 'sandero_1027_gao__port_506198', 0.0014584699, 332, '355'), ('916235064', '3008_1027_gao__port_506385', 0.005645383, 332, '355'), ('916235064', 'q50_1027_gao__port_506239', 0.001116375, 332, '355'), ('916235064', 'latitude_1027_gao__port_506236', 0.0008018051, 332, '355'), ('916235064', 'v40_1027_gao__port_506391', 0.001714542, 332, '355'), ('916235064', 'xsara_1027_gao__port_506087', 0.0009821726, 332, '355'), ('916235064', 'grand_c_max_1027_gao__port_506342', 0.0017954878, 332, '355'), ('916235064', 'swift_1027_gao__port_506149', 0.0015018679, 332, '355'), ('916235064', 'serie_1_1027_gao__port_506184', 0.0015139927, 332, '355'), ('916235064', 'xc70_1027_gao__port_506393', 0.0036189347, 332, '355'), ('916235064', 'master_1027_gao__port_506203', 0.007958394, 332, '355'), ('916235064', 'clio_1027_gao__port_506280', 0.0029577191, 332, '355'), ('916235064', 'duster_1027_gao__port_506216', 0.0007443036, 332, '355'), ('916235064', 'traveller_1027_gao__port_506403', 0.004293621, 332, '355'), ('916235064', 'tipo_1027_gao__port_506355', 0.0010928999, 332, '355'), ('916235064', 'rav_4_1027_gao__port_506332', 0.0013602059, 332, '355'), ('916235064', 'coccinelle_1027_gao__port_506259', 0.0034948604, 332, '355'), ('916235064', 'spacetourer_1027_gao__port_506401', 0.0030969484, 332, '355'), ('916235064', 'xe_1027_gao__port_506357', 0.0014471222, 332, '355'), ('916235064', 'ds3_1027_gao__port_506324', 0.0013092074, 332, '355'), ('916235064', 'mx_5_1027_gao__port_506098', 0.002588833, 332, '355'), ('916235064', 'land_cruiser_1027_gao__port_506315', 0.009532341, 332, '355'), ('916235064', 'classe_b_1027_gao__port_506335', 0.0017213442, 332, '355'), ('916235064', '806_1027_gao__port_506088', 0.002561532, 332, '355'), ('916235064', 'rx_8_1027_gao__port_506046', 0.0036226173, 332, '355'), ('916235064', 'spark_1027_gao__port_506185', 0.0010075701, 332, '355'), ('916235064', '6_1027_gao__port_506171', 0.0011181624, 332, '355'), ('916235064', 'bravo_1027_gao__port_506080', 0.0014650921, 332, '355'), ('916235064', 'nx_1027_gao__port_506345', 0.0013683256, 332, '355'), ('916235064', 'sharan_1027_gao__port_506347', 0.0050922018, 332, '355'), ('916235064', 'x_type_1027_gao__port_506067', 0.00078010786, 332, '355'), ('916235064', 'jimny_1027_gao__port_506233', 0.0046066716, 332, '355'), ('916235064', 'wrangler_1027_gao__port_506225', 0.001799802, 332, '355'), ('916235064', 'c_crosser_1027_gao__port_506312', 0.0015925592, 332, '355'), ('916235064', 'v70_1027_gao__port_506278', 0.0019675149, 332, '355'), ('916235064', 'classe_e_1027_gao__port_506300', 0.0017368586, 332, '355'), ('916235064', 'classe_v_1027_gao__port_506258', 0.012731758, 332, '355'), ('916235064', 'm3_1027_gao__port_506182', 0.0023373484, 332, '355'), ('916235064', 'abarth_500_1027_gao__port_506226', 0.0040434003, 332, '355'), ('916235064', 'serie_6_1027_gao__port_506262', 0.0011314218, 332, '355'), ('916235064', 'modus_1027_gao__port_506146', 0.001829209, 332, '355'), ('916235064', '3_1027_gao__port_506113', 0.0015082227, 332, '355'), ('916235064', '405_1027_gao__port_506108', 0.008055673, 332, '355'), ('916235064', 'allroad_1027_gao__port_506297', 0.0010596248, 332, '355'), ('916235064', 'auris_1027_gao__port_506322', 0.0011522867, 332, '355'), ('916235064', 'galaxy_1027_gao__port_506143', 0.0025144962, 332, '355'), ('916235064', 'giulietta_1027_gao__port_506363', 0.00086504366, 332, '355'), ('916235064', '106_1027_gao__port_506073', 0.00827062, 332, '355'), ('916235064', 'classe_m_1027_gao__port_506154', 0.003001643, 332, '355'), ('916235064', 'espace_1027_gao__port_506313', 0.0010643916, 332, '355'), ('916235064', 'panda_1027_gao__port_506189', 0.009030353, 332, '355'), ('916235064', 'rcz_1027_gao__port_506197', 0.0011292502, 332, '355'), ('916235064', '4007_1027_gao__port_506162', 0.0006792294, 332, '355'), ('916235064', 'classe_cl_1027_gao__port_506249', 0.0010859325, 332, '355'), ('916235064', 'leaf_1027_gao__port_506139', 0.0018037524, 332, '355'), ('916235064', 'octavia_1027_gao__port_506237', 0.0018601292, 332, '355'), ('916235064', 'ds4_1027_gao__port_506336', 0.0024154065, 332, '355'), ('916235064', 'freelander_1027_gao__port_506084', 0.0023471802, 332, '355'), ('916235064', 'evasion_1027_gao__port_506109', 0.0031134957, 332, '355'), ('916235064', 'punto_1027_gao__port_506106', 0.0019493502, 332, '355'), ('916235064', '2cv_1027_gao__port_506045', 0.007974933, 332, '355'), ('916235064', 'x4_1027_gao__port_506392', 0.0017947983, 332, '355'), ('916235064', 'antara_1027_gao__port_506247', 0.0012467758, 332, '355'), ('916235064', 'murano_1027_gao__port_506316', 0.00060880015, 332, '355'), ('916235064', 'alto_1027_gao__port_506201', 0.009231896, 332, '355'), ('916235064', 'meriva_1027_gao__port_506353', 0.0013761518, 332, '355'), ('916235064', 'orlando_1027_gao__port_506305', 0.0018459162, 332, '355'), ('916235064', 'new_beetle_1027_gao__port_506050', 0.0011637013, 332, '355'), ('916235064', '306_1027_gao__port_506145', 0.0035068155, 332, '355'), ('916235064', 'tiguan_1027_gao__port_506362', 0.0026824567, 332, '355'), ('916235064', 's_type_1027_gao__port_506101', 0.0011381401, 332, '355'), ('916235064', 'c1_1027_gao__port_506128', 0.0027513453, 332, '355'), ('916235064', 'vectra_1027_gao__port_506044', 0.0011988143, 332, '355'), ('916235064', 'outlander_1027_gao__port_506317', 0.001712139, 332, '355'), ('916235064', '307_1027_gao__port_506074', 0.0020009007, 332, '355'), ('916235064', 'a6_s6_1027_gao__port_506134', 0.001657052, 332, '355'), ('916235064', 'nemo_combi_1027_gao__port_506196', 0.0022661076, 332, '355'), ('916235064', 'berlingo_1027_gao__port_506194', 0.0046628197, 332, '355'), ('916235064', 'partner_1027_gao__port_506285', 0.0039412924, 332, '355'), ('916235064', 'cayenne_1027_gao__port_506177', 0.003797821, 332, '355'), ('916235064', 'quattroporte_1027_gao__port_506240', 0.002444018, 332, '355'), ('916235064', 'c_max_1027_gao__port_506282', 0.0013122569, 332, '355'), ('916235064', 'fabia_1027_gao__port_506396', 0.0052995775, 332, '355'), ('916235064', 'cx_3_1027_gao__port_506281', 0.0014462185, 332, '355'), ('916235064', 'x_trail_1027_gao__port_506264', 0.0018314207, 332, '355'), ('916235064', 'scirocco_1027_gao__port_506276', 0.0047903094, 332, '355'), ('916235064', 'matiz_1027_gao__port_506144', 0.0017559103, 332, '355'), ('916235064', 'tigra_1027_gao__port_506069', 0.0008541954, 332, '355'), ('916235064', 'escort_1027_gao__port_506091', 0.004840247, 332, '355'), ('916235064', 'c2_1027_gao__port_506081', 0.001490288, 332, '355'), ('916235064', 'mini_1027_gao__port_506168', 0.0011920253, 332, '355'), ('916235064', 'i30_1027_gao__port_506291', 0.0006324606, 332, '355'), ('916235064', 'picanto_1027_gao__port_506238', 0.0029748445, 332, '355'), ('916235064', 'mito_1027_gao__port_506072', 0.0015078369, 332, '355'), ('916235064', 'impreza_1027_gao__port_506085', 0.002019112, 332, '355'), ('916235064', 'kangoo_1027_gao__port_506235', 0.006584493, 332, '355'), ('916235064', 'a4_1027_gao__port_506193', 0.001987239, 332, '355'), ('916235064', 'cayman_1027_gao__port_506268', 0.0018139802, 332, '355'), ('916235064', 'sportage_1027_gao__port_506148', 0.0014273935, 332, '355'), ('916235064', 'up_1027_gao__port_506356', 0.0068637016, 332, '355'), ('916235064', 'optima_1027_gao__port_506386', 0.00089173636, 332, '355'), ('916235064', 'defender_1027_gao__port_506229', 0.006722186, 332, '355'), ('916235064', 'serie_2_1027_gao__port_506256', 0.002227326, 332, '355'), ('916235064', 'edge_1027_gao__port_506187', 0.00087463163, 332, '355'), ('916235064', 'r19_1027_gao__port_506110', 0.004942841, 332, '355'), ('916235064', 'jetta_1027_gao__port_506304', 0.0036195705, 332, '355'), ('916235064', 'eos_1027_gao__port_506115', 0.0038923176, 332, '355'), ('916235064', 'accord_1027_gao__port_506214', 0.0020126875, 332, '355'), ('916235064', 'yaris_1027_gao__port_506334', 0.0032329976, 332, '355'), ('916235064', 'classe_cls_1027_gao__port_506289', 0.0007851111, 332, '355'), ('916235064', 'polo_1027_gao__port_506361', 0.0043107197, 332, '355'), ('916235064', 'serie_4_1027_gao__port_506366', 0.0011476217, 332, '355'), ('916235064', 'mini_cabriolet_1027_gao__port_506204', 0.0008377132, 332, '355'), ('916235064', 'prius_1027_gao__port_506190', 0.0011488078, 332, '355'), ('916235064', 'lodgy_1027_gao__port_506188', 0.0020171355, 332, '355'), ('916235064', 'serie_7_1027_gao__port_506307', 0.0012477492, 332, '355'), ('916235064', 'c15_1027_gao__port_506055', 0.01771508, 332, '355'), ('916235064', 'kadjar_1027_gao__port_506389', 0.0012504428, 332, '355'), ('916235064', 'insignia_1027_gao__port_506364', 0.0016431045, 332, '355'), ('916235064', '308_1027_gao__port_506279', 0.002123339, 332, '355'), ('916235064', 'roomster_1027_gao__port_506241', 0.0018012933, 332, '355'), ('916235064', '80_1027_gao__port_506057', 0.0046136877, 332, '355'), ('916235064', '309_1027_gao__port_506063', 0.013526396, 332, '355'), ('916235064', 'tucson_1027_gao__port_506320', 0.0021234627, 332, '355'), ('916235064', 'x3_1027_gao__port_506212', 0.00089765183, 332, '355'), ('916235064', 'xf_1027_gao__port_506263', 0.0011166844, 332, '355'), ('916235064', '2008_1027_gao__port_506394', 0.0026409323, 332, '355'), ('916235064', 'passat_1027_gao__port_506306', 0.0014975086, 332, '355'), ('916235064', 'compass_1027_gao__port_506260', 0.0032557966, 332, '355'), ('916235064', 'twingo_1027_gao__port_506309', 0.0064986506, 332, '355'), ('916235064', 'micra_1027_gao__port_506221', 0.0035860348, 332, '355'), ('916235064', 'golf_1027_gao__port_506155', 0.0031932737, 332, '355'), ('916235064', 'soul_1027_gao__port_506176', 0.0012888586, 332, '355'), ('916235064', 'rapid_1027_gao__port_506398', 0.0025915466, 332, '355'), ('916235064', 'forester_1027_gao__port_506360', 0.0022760385, 332, '355'), ('916235064', 'slk_1027_gao__port_506210', 0.0015844845, 332, '355'), ('916235064', 'forfour_1027_gao__port_506341', 0.0021795542, 332, '355'), ('916235064', 'serie_5_1027_gao__port_506209', 0.0013695179, 332, '355'), ('916235064', 'xj_1027_gao__port_506170', 0.002601203, 332, '355'), ('916235064', 'pajero_1027_gao__port_506097', 0.005211227, 332, '355'), ('916235064', 'agila_1027_gao__port_506119', 0.004848958, 332, '355'), ('916235064', 'a6_1027_gao__port_506163', 0.0018910117, 332, '355'), ('916235064', 'fox_1027_gao__port_506092', 0.0008464996, 332, '355'), ('916235064', 'boxster_1027_gao__port_506267', 0.0015941721, 332, '355'), ('916235064', 'altea_1027_gao__port_506246', 0.0021390466, 332, '355'), ('916235064', 'samurai_1027_gao__port_506047', 0.006251647, 332, '355'), ('916235064', 'trax_1027_gao__port_506296', 0.0019435452, 332, '355'), ('916235064', 'getz_1027_gao__port_506058', 0.0016383227, 332, '355'), ('916235064', 'cherokee_1027_gao__port_506269', 0.0029806844, 332, '355'), ('916235064', 'koleos_1027_gao__port_506378', 0.0015590898, 332, '355'), ('916235064', 'z_series_1027_gao__port_506123', 0.0016563536, 332, '355'), ('916235064', 'ecosport_1027_gao__port_506271', 0.0013229419, 332, '355'), ('916235064', 'space_star_1027_gao__port_506277', 0.0021140124, 332, '355'), ('916235064', 'rs3_sportback_1027_gao__port_506207', 0.0019114943, 332, '355'), ('916235064', 'civic_1027_gao__port_506141', 0.0026896456, 332, '355'), ('916235064', 'talisman_1027_gao__port_506390', 0.0007612969, 332, '355'), ('916235064', 'f_pace_1027_gao__port_506314', 0.0016165051, 332, '355'), ('916235064', 'classe_c_1027_gao__port_506299', 0.0017940501, 332, '355'), ('916235064', 'tt_1027_gao__port_506075', 0.0013934447, 332, '355'), ('916235064', 'pathfinder_1027_gao__port_506183', 0.0016514092, 332, '355'), ('916235064', '156_1027_gao__port_506157', 0.0015443967, 332, '355'), ('916235064', 'cx_5_1027_gao__port_506228', 0.0014413124, 332, '355'), ('916235064', 'scenic_1027_gao__port_506255', 0.0016080706, 332, '355'), ('916235064', 'yeti_1027_gao__port_506358', 0.002091413, 332, '355'), ('916235064', 'mustang_1027_gao__port_506053', 0.010053302, 332, '355'), ('916235064', 'stilo_1027_gao__port_506060', 0.0010831376, 332, '355'), ('916235064', 'ateca_1027_gao__port_506382', 0.0017008726, 332, '355'), ('916235064', 'fiorino_1027_gao__port_506217', 0.009200124, 332, '355'), ('916235064', 'classe_glk_1027_gao__port_506290', 0.0017015906, 332, '355'), ('916235064', 'fortwo_1027_gao__port_506230', 0.0016008316, 332, '355'), ('916235064', 'cruze_1027_gao__port_506186', 0.0010051767, 332, '355'), ('916235064', '107_1027_gao__port_506213', 0.0016275843, 332, '355'), ('916235064', 'aygo_1027_gao__port_506248', 0.0032435178, 332, '355'), ('916235064', 'rx_1027_gao__port_506354', 0.0010631674, 332, '355'), ('916235064', '500_1027_gao__port_506245', 0.0016354468, 332, '355'), ('916235064', 'bora_1027_gao__port_506104', 0.0038168954, 332, '355'), ('916235064', 'transit_1027_gao__port_506111', 0.0048620747, 332, '355'), ('916235064', 'pt_cruiser_1027_gao__port_506054', 0.0019161776, 332, '355'), ('916235064', 'patrol_1027_gao__port_506068', 0.0042399974, 332, '355'), ('916235064', 'r8_1027_gao__port_506156', 0.0012722366, 332, '355'), ('916235064', 'xm_1027_gao__port_506102', 0.0022673472, 332, '355'), ('916235064', 's60_1027_gao__port_506191', 0.0031989103, 332, '355'), ('916235064', 'aveo_1027_gao__port_506158', 0.0038400472, 332, '355'), ('916235064', 'captiva_1027_gao__port_506159', 0.0017186139, 332, '355'), ('916235064', 'ax_1027_gao__port_506153', 0.006895962, 332, '355'), ('916235064', 'rexton_1027_gao__port_506107', 0.0013019991, 332, '355'), ('916235064', 'camaro_1027_gao__port_506056', 0.0024906942, 332, '355'), ('916235064', 'ypsilon_1027_gao__port_506131', 0.00195405, 332, '355'), ('916235064', 'delta_1027_gao__port_506165', 0.0014003292, 332, '355'), ('916235064', 'c4_1027_gao__port_506370', 0.0013005522, 332, '355'), ('916235064', 'zx_1027_gao__port_506161', 0.004594108, 332, '355'), ('916235064', 'verso_1027_gao__port_506242', 0.0007719758, 332, '355'), ('916235064', 'superb_1027_gao__port_506327', 0.0019941824, 332, '355'), ('916235064', 'r5_1027_gao__port_506253', 0.009546707, 332, '355'), ('916235064', 'caddy_1027_gao__port_506330', 0.01382874, 332, '355'), ('916235064', 'x5_1027_gao__port_506243', 0.0011203574, 332, '355'), ('916235064', 'f_type_1027_gao__port_506231', 0.00082987954, 332, '355'), ('916235064', 'fusion_1027_gao__port_506096', 0.0012667527, 332, '355'), ('916235064', 'dokker_1027_gao__port_506331', 0.005357473, 332, '355'), ('916235064', '205_1027_gao__port_506062', 0.0066863173, 332, '355'), ('916235064', 'macan_1027_gao__port_506195', 0.0015572207, 332, '355'), ('916235064', 'tourneo_1027_gao__port_506369', 0.00640061, 332, '355'), ('916235064', '108_1027_gao__port_506384', 0.005263989, 332, '355'), ('916235064', '9_3_1027_gao__port_506071', 0.0008373739, 332, '355'), ('916235064', 'mondeo_1027_gao__port_506116', 0.0014394012, 332, '355'), ('916235064', 'cr_v_1027_gao__port_506164', 0.0016412234, 332, '355'), ('916235064', 'c30_1027_gao__port_506090', 0.0017484056, 332, '355'), ('916235064', 'pulsar_1027_gao__port_506397', 0.0012017348, 332, '355'), ('916235064', 'ibiza_1027_gao__port_506273', 0.003723159, 332, '355'), ('916235064', 'a1_1027_gao__port_506338', 0.0012344852, 332, '355'), ('916235064', 'matrix_1027_gao__port_506140', 0.00070758583, 332, '355'), ('916235064', 'carnival_1027_gao__port_506136', 0.002280904, 332, '355'), ('916235064', 'xantia_1027_gao__port_506086', 0.0021959264, 332, '355'), ('916235064', 'terrano_1027_gao__port_506083', 0.0020292788, 332, '355'), ('916235064', 'q3_1027_gao__port_506275', 0.0011263487, 332, '355'), ('916235064', 'hr_v_1027_gao__port_506283', 0.0017802508, 332, '355'), ('916235064', 'expert_1027_gao__port_506142', 0.0073694875, 332, '355'), ('916235064', 'multivan_1027_gao__port_506383', 0.00650532, 332, '355'), ('916235064', 'venga_1027_gao__port_506380', 0.0008003971, 332, '355'), ('916235064', 'scudo_1027_gao__port_506129', 0.0055924854, 332, '355'), ('916235064', 'laguna_1027_gao__port_506368', 0.0007133706, 332, '355'), ('916235064', 'vel_satis_1027_gao__port_506130', 0.002726076, 332, '355'), ('916235064', 'b_max_1027_gao__port_506367', 0.0017243357, 332, '355'), ('916235064', 'ignis_1027_gao__port_506292', 0.004356217, 332, '355'), ('916235064', '159_1027_gao__port_506064', 0.0010782072, 332, '355'), ('916235064', 'grande_punto_1027_gao__port_506138', 0.0023632485, 332, '355'), ('916235064', 'logan_1027_gao__port_506167', 0.004397134, 332, '355'), ('916235064', 's_max_1027_gao__port_506223', 0.001252343, 332, '355'), ('916235064', 'caravelle_1027_gao__port_506351', 0.003029375, 332, '355'), ('916235064', 'adam_1027_gao__port_506079', 0.0010536193, 332, '355'), ('916235064', '406_1027_gao__port_506132', 0.0013574458, 332, '355'), ('916235064', 'q30_1027_gao__port_506293', 0.00097139477, 332, '355'), ('916235064', 'almera_1027_gao__port_506089', 0.0010238016, 332, '355'), ('916235064', 'corsa_1027_gao__port_506095', 0.0025198879, 332, '355'), ('916235064', 'corolla_1027_gao__port_506120', 0.0026819613, 332, '355'), ('916235064', 'xc60_1027_gao__port_506388', 0.0018984496, 332, '355'), ('916235064', 'viano_1027_gao__port_506211', 0.0026940526, 332, '355'), ('916235064', 'pro_cee_d_1027_gao__port_506274', 0.0008319524, 332, '355'), ('916235064', 'a3_1027_gao__port_506321', 0.0037383058, 332, '355'), ('916235064', 'v50_1027_gao__port_506150', 0.00079182506, 332, '355'), ('916235064', 'voyager_1027_gao__port_506169', 0.0030522551, 332, '355'), ('916235064', 'corvette_1027_gao__port_506049', 0.0037236721, 332, '355'), ('916235064', 'rio_1027_gao__port_506379', 0.001773744, 332, '355'), ('916235064', 'jazz_1027_gao__port_506252', 0.0015302772, 332, '355'), ('916235064', '200_1027_gao__port_506112', 0.0040874984, 332, '355'), ('916235064', 'tts_1027_gao__port_506199', 0.0011862821, 332, '355'), ('916235064', 'zafira_1027_gao__port_506287', 0.0026946918, 332, '355'), ('916235064', 'asx_1027_gao__port_506266', 0.0011406197, 332, '355'), ('916235064', '607_1027_gao__port_506118', 0.0012526207, 332, '355'), ('916235064', '207_1027_gao__port_506103', 0.0015147214, 332, '355'), ('916235064', 'classe_s_1027_gao__port_506301', 0.00316558, 332, '355'), ('916235064', 'c6_1027_gao__port_506105', 0.001734736, 332, '355'), ('916235064', 'express_1027_gao__port_506137', 0.016730625, 332, '355'), ('916235064', 'classe_gla_1027_gao__port_506352', 0.0018254801, 332, '355'), ('916235064', 'v60_1027_gao__port_506333', 0.0021457663, 332, '355'), ('916235064', 'ka_1027_gao__port_506180', 0.001415164, 332, '355'), ('916235064', 'range_rover_1027_gao__port_506254', 0.002055093, 332, '355'), ('916235064', 'discovery_1027_gao__port_506375', 0.0022963856, 332, '355'), ('916235064', 'classe_r_1027_gao__port_506270', 0.0013942117, 332, '355'), ('916235064', 'transporter_1027_gao__port_506319', 0.0119707845, 332, '355'), ('916235064', 'cee_d_1027_gao__port_506288', 0.001054702, 332, '355'), ('916235064', 'zoe_1027_gao__port_506244', 0.0020712344, 332, '355'), ('916235064', 'i20_1027_gao__port_506284', 0.0017869547, 332, '355'), ('916235064', 'gtv_1027_gao__port_506059', 0.00572377, 332, '355'), ('916235064', 's4_avant_1027_gao__port_506261', 0.0027664162, 332, '355'), ('916235064', 'x1_1027_gao__port_506372', 0.0017145447, 332, '355'), ('916235064', 'autres_1027_gao__port_506127', 0.004825847, 332, '355'), ('916235064', '208_1027_gao__port_506359', 0.0018684062, 332, '355'), ('916235064', 'c8_1027_gao__port_506135', 0.0012577998, 332, '355'), ('916235064', 'astra_1027_gao__port_506215', 0.0012624015, 332, '355'), ('916235064', '2_1027_gao__port_506151', 0.0009245341, 332, '355'), ('916235064', 'doblo_1027_gao__port_506251', 0.007466577, 332, '355'), ('916235064', '807_1027_gao__port_506152', 0.00072884944, 332, '355'), ('916235064', '206_1027_gao__port_506126', 0.0010384651, 332, '355'), ('916235064', 'a7_1027_gao__port_506373', 0.00069110165, 332, '355'), ('916235064', 'renegade_1027_gao__port_506346', 0.00214163, 332, '355')]]} begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 7.3909759521484375e-06 save missing photos in datou_result : time spend for datou_step_exec : 26.081963062286377 time spend to save output : 2.1153767108917236 total time spend for step 1 : 28.1973397731781 step2:argmax Thu Feb 27 00:37:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740613033_1084700_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1740613033_1084700_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step Argmax ! calculate argmax for thcl : 355 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True photo_id : 916235064 output[photo_id] : [('916235064', 'c15_1027_gao__port_506055', 0.01771508, 332, '355'), 'temp/1740613033_1084700_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg'] begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 1 insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) first line : ('916235064', '2049863950', '332') ... last line : ('916235064', '2049863950', '332') time used for this insertion : 0.012870311737060547 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 1 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 0.019052505493164062 len list_finale : 1, len picture : 1 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('2', None, '916235064', 'c15_1027_gao__port_506055', None, None, '2049863950', '0.01771508', None)] time used for this insertion : 0.017994165420532227 saving photo_ids in datou_result photo id not in port begin to insert list_values into mtr_datou_result : length of list_values in save_final : 0 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [] time used for this insertion : 4.291534423828125e-06 save missing photos in datou_result : time spend for datou_step_exec : 0.003989696502685547 time spend to save output : 0.05048489570617676 total time spend for step 2 : 0.054474592208862305 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : {'916235064': [('916235064', 'c15_1027_gao__port_506055', 0.01771508, 332, '355'), 'temp/1740613033_1084700_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg']} ############################### TEST tfhub2 ################################ TEST TFHUB2 ######################## test with use_multi_inputs=0 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4567 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4567 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4567 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4567 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12835 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12836 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171252784,1171252764,1171252487) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171252487 begin to download photo : 1171252764 begin to download photo : 1171252784 download finish for photo 1171252784 download finish for photo 1171252764 download finish for photo 1171252487 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.18418431282043457 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Thu Feb 27 00:37:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740613062_1084700_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg': 1171252784, 'temp/1740613062_1084700_1171252764_29d5179a892cc50aadc9d67245534b59.jpg': 1171252764, 'temp/1740613062_1084700_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg': 1171252487} map_photo_id_path_extension : {1171252784: {'path': 'temp/1740613062_1084700_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg', 'extension': 'jpg'}, 1171252764: {'path': 'temp/1740613062_1084700_1171252764_29d5179a892cc50aadc9d67245534b59.jpg', 'extension': 'jpg'}, 1171252487: {'path': 'temp/1740613062_1084700_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3609': 1} we are using the classfication for only one thcl 3609 begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-02-27 00:37:51.498048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-27 00:37:51.498755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:37:51.498886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:37:51.498980: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:37:51.501785: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:37:51.501876: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:37:51.505892: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:37:51.507497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:37:51.514832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:37:51.516036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:37:51.516507: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-27 00:37:51.530698: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-27 00:37:51.532056: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff380000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-27 00:37:51.532077: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-27 00:37:51.534702: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4a62bb20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-27 00:37:51.534721: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-27 00:37:51.535402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 00:37:51.535499: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:37:51.535519: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 00:37:51.535591: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 00:37:51.535614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 00:37:51.535648: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 00:37:51.535681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 00:37:51.535713: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 00:37:51.536517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 00:37:51.536561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 00:37:51.536600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 00:37:51.536611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 00:37:51.536623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 00:37:51.537486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2646 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3609 To do loadFromThcl(), then load ParamDescType : thcl3609 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3609) thcls : [{'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'}] thcl {'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'} Update svm_hashtag_type_desc : 5832 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5832) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5832, 'tfhub_19_06_2023', 1280, 1280, 'tfhub_19_06_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 6, 19, 12, 55, 22), datetime.datetime(2023, 6, 19, 12, 55, 22)) model_name : tfhub_19_06_2023 model_param file didn't exist model_name : tfhub_19_06_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-02-27 00:37:58.765550: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-27 00:37:58.766151: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/tfhub_19_06_2023 /data/models_weight/tfhub_19_06_2023/Confusion_Matrix.png size_local : 57753 size in s3 : 57753 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_jrm.jpg size_local : 79724 size in s3 : 79724 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcm.jpg size_local : 83556 size in s3 : 83556 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcnc.jpg size_local : 74107 size in s3 : 74107 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pehd.jpg size_local : 72705 size in s3 : 72705 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_tapis_vide.jpg size_local : 70874 size in s3 : 70874 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 checkpoint already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216488 size in s3 : 216488 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:21 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_weights.h5 size_local : 16499144 size in s3 : 16499144 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:15 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 77, in create_tfhub_model hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module"), File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171252784, 1171252764, 1171252487] map_info['map_portfolio_photo'] : {} final : True mtd_id 4567 list_pids : [1171252784, 1171252764, 1171252487] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4567', None, '1171252784', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252764', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252487', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.020017385482788086 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 16.8969669342041 time spend to save output : 0.024937868118286133 total time spend for step 0 : 16.921904802322388 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171252784': [(1171252784, 'jrm', 0.9677492, 4674, '3609'), 'temp/1687511175_1882837_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg'], '1171252764': [(1171252764, 'jrm', 0.9853587, 4674, '3609'), 'temp/1687511175_1882837_1171252764_29d5179a892cc50aadc9d67245534b59.jpg'], '1171252487': [(1171252487, 'jrm', 0.9262757, 4674, '3609'), 'temp/1687511175_1882837_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg']} got : None ######################## test with use_multi_inputs=1 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4621 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4621 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4621 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4621 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12927 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12928 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171291875,1171275372,1171275314) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171275314 begin to download photo : 1171275372 begin to download photo : 1171291875 download finish for photo 1171275372 download finish for photo 1171291875 download finish for photo 1171275314 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.22822880744934082 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Thu Feb 27 00:37:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740613079_1084700_1171275372_76d81364ff7df843bff095f45c07ba35.jpg': 1171275372, 'temp/1740613079_1084700_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg': 1171291875, 'temp/1740613079_1084700_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg': 1171275314} map_photo_id_path_extension : {1171275372: {'path': 'temp/1740613079_1084700_1171275372_76d81364ff7df843bff095f45c07ba35.jpg', 'extension': 'jpg'}, 1171291875: {'path': 'temp/1740613079_1084700_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg', 'extension': 'jpg'}, 1171275314: {'path': 'temp/1740613079_1084700_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3655': 1} we are using the classfication for only one thcl 3655 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 32 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 32 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 32 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 32 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 32 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 32 wait 20 seconds l 3637 free memory gpu now : 32 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3655 To do loadFromThcl(), then load ParamDescType : thcl3655 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3655) thcls : [{'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'}] thcl {'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'} Update svm_hashtag_type_desc : 5862 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5862) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5862, 'tfhub_18_7_2023', 1280, 1280, 'tfhub_18_7_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 7, 18, 22, 46, 29), datetime.datetime(2023, 7, 18, 22, 46, 29)) model_name : tfhub_18_7_2023 model_param file didn't exist model_name : tfhub_18_7_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/tfhub_18_7_2023 /data/models_weight/tfhub_18_7_2023/Confusion_Matrix.png size_local : 54360 size in s3 : 54360 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:28 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_jrm.jpg size_local : 72583 size in s3 : 72583 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcm.jpg size_local : 81681 size in s3 : 81681 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcnc.jpg size_local : 79510 size in s3 : 79510 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pehd.jpg size_local : 59936 size in s3 : 59936 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_tapis_vide.jpg size_local : 78974 size in s3 : 78974 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 checkpoint already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216529 size in s3 : 216529 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279748 size in s3 : 32279748 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_weights.h5 size_local : 16500868 size in s3 : 16500868 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:18 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 62, in create_tfhub_model fe_layer = hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module", File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171275372, 1171291875, 1171275314] map_info['map_portfolio_photo'] : {} final : True mtd_id 4621 list_pids : [1171275372, 1171291875, 1171275314] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4621', None, '1171275372', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171291875', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171275314', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.013579607009887695 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 133.56652688980103 time spend to save output : 0.014027118682861328 total time spend for step 0 : 133.5805540084839 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171291875': [(1171291875, 'tapis_vide', 0.97062814, 4723, '3655'), 'temp/1691745841_1143057_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg'], '1171275372': [(1171275372, 'tapis_vide', 0.9674145, 4723, '3655'), 'temp/1691745841_1143057_1171275372_76d81364ff7df843bff095f45c07ba35.jpg'], '1171275314': [(1171275314, 'tapis_vide', 0.96509415, 4723, '3655'), 'temp/1691745841_1143057_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg']} got : None ERROR tfhub2 FAILED ############################### TEST ordonner ################################ To do loadFromThcl(), then load ParamDescType : thcl358 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (358) thcls : [{'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3392 ['FirstUploadExperveo_vignette__port_505674', 'CAR_EXTERIEUR_Roue__port_503398', 'FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486', 'FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465', 'CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198', 'CAR_EXTERIEUR_Face_avant_axe_droit__port_504451', 'CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235', 'FirstUploadExperveo_vin__port_505675', 'CAR_EXTERIEUR_cote_droite__port_504108', 'CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565', 'FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201', 'cartegrise_orientation__port_505064', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217', 'CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531', 'CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218', 'CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214', 'CAR_EXTERIEUR_Angle_avant_droit__port_504087', 'FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484', 'CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563', 'CAR_EXTERIEUR_Angle_arriere_droit__port_504160', 'CAR_EXTERIEUR_arriere__port_504184', 'CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562', 'INTERIEUR_Compteur_kilometrique__port_503644', 'CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494', 'CAR_EXTERIEUR_Angle_arriere_gauche__port_504170', 'CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226', 'CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202', 'CAR_EXTERIEUR_moteur__port_503704', 'FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487', 'CAR_INTERIEUR_siege_arriere_class_1__port_506551', 'CAR_EXTERIEUR_avant__port_504146', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215', 'CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225', 'CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564', 'FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'CAR_INTERIEUR_coffre__port_503412', 'FirstUploadExperveo_rouetranche__port_505677', 'UploadPhotoImmatBest_class_1__port_505051', 'CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532', 'CAR_EXTERIEUR_angle_avant_gauche__port_504098', 'CAR_EXTERIEUR_face_avant_axe_gauche__port_504236', 'CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540', 'CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233', 'CAR_EXTERIEUR_roue_de_secour__port_503763', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199', 'CAR_EXTERIEUR_cote_gauche__port_504017', 'CAR_INTERIEUR_avant_volant_class_1__port_506503', 'CAR_INTERIEUR_avant_volant_class_2__port_506504', 'CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'] 51 SELECT hashtag_id,hashtag FROM MTRBack.hashtags where hashtag in ('FirstUploadExperveo_vignette__port_505674','CAR_EXTERIEUR_Roue__port_503398','FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486','FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485','CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465','CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198','CAR_EXTERIEUR_Face_avant_axe_droit__port_504451','CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235','FirstUploadExperveo_vin__port_505675','CAR_EXTERIEUR_cote_droite__port_504108','CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565','FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483','CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201','cartegrise_orientation__port_505064','CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217','CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531','CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218','CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214','CAR_EXTERIEUR_Angle_avant_droit__port_504087','FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484','CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563','CAR_EXTERIEUR_Angle_arriere_droit__port_504160','CAR_EXTERIEUR_arriere__port_504184','CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562','INTERIEUR_Compteur_kilometrique__port_503644','CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494','CAR_EXTERIEUR_Angle_arriere_gauche__port_504170','CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226','CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202','CAR_EXTERIEUR_moteur__port_503704','FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487','CAR_INTERIEUR_siege_arriere_class_1__port_506551','CAR_EXTERIEUR_avant__port_504146','CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215','CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225','CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564','FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482','CAR_INTERIEUR_coffre__port_503412','FirstUploadExperveo_rouetranche__port_505677','UploadPhotoImmatBest_class_1__port_505051','CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532','CAR_EXTERIEUR_angle_avant_gauche__port_504098','CAR_EXTERIEUR_face_avant_axe_gauche__port_504236','CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540','CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233','CAR_EXTERIEUR_roue_de_secour__port_503763','CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199','CAR_EXTERIEUR_cote_gauche__port_504017','CAR_INTERIEUR_avant_volant_class_1__port_506503','CAR_INTERIEUR_avant_volant_class_2__port_506504','CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'); 51 dict_keys(['cartegrise_orientation__port_505064', 'car_exterieur_angle_arriere_droit_axe_arriere__port_504217', 'car_exterieur_angle_arriere_droit_axe_droit__port_504215', 'car_exterieur_angle_arriere_droit__port_504160', 'car_exterieur_angle_arriere_gauche_axe_arriere__port_504201', 'car_exterieur_angle_arriere_gauche_axe_gauche__port_504199', 'car_exterieur_angle_arriere_gauche__port_504170', 'car_exterieur_angle_avant_droit_axe_arriere__port_504226', 'car_exterieur_angle_avant_droit_axe_droit__port_504225', 'car_exterieur_angle_avant_droit__port_504087', 'car_exterieur_angle_avant_gauche_axe_avant__port_504235', 'car_exterieur_angle_avant_gauche_axe_gauche__port_504234', 'car_exterieur_angle_avant_gauche__port_504098', 'car_exterieur_arriere__port_504184', 'car_exterieur_avant__port_504146', 'car_exterieur_cote_droite__port_504108', 'car_exterieur_cote_droit_axe_arriere__port_504214', 'car_exterieur_cote_droit_axe_avant__port_504465', 'car_exterieur_cote_gauche_axe_arriere__port_504198', 'car_exterieur_cote_gauche_axe_avant__port_504233', 'car_exterieur_cote_gauche__port_504017', 'car_exterieur_face_arriere_axe_droit__port_504218', 'car_exterieur_face_arriere_axe_gauche__port_504202', 'car_exterieur_face_avant_axe_droit__port_504451', 'car_exterieur_face_avant_axe_gauche__port_504236', 'car_exterieur_moteur__port_503704', 'car_exterieur_roue_de_secour__port_503763', 'car_exterieur_roue__port_503398', 'car_interieur_avant_volant_class_1__port_506503', 'car_interieur_avant_volant_class_2__port_506504', 'car_interieur_avant_volant_class_6_boutonrond__port_506562', 'car_interieur_avant_volant_class_6_class_2__port_506563', 'car_interieur_avant_volant_class_6_ecrangrosplan__port_506564', 'car_interieur_avant_volant_class_6_levierdevitesse__port_506565', 'car_interieur_avant_vue-arriere_class_1__port_506531', 'car_interieur_avant_vue-arriere_class_2__port_506532', 'car_interieur_avant_vue_droite_habitacle_class_1__port_506540', 'car_interieur_avant_vue_gauche_habitacle_class_1__port_506494', 'car_interieur_coffre__port_503412', 'car_interieur_siege_arriere_class_1__port_506551', 'firstuploadexperveo_carrosseriegrosplan_carrosserie__port_506483', 'firstuploadexperveo_carrosseriegrosplan_class_6__port_506487', 'firstuploadexperveo_carrosseriegrosplan_morceauderoue__port_506484', 'firstuploadexperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'firstuploadexperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'firstuploadexperveo_carrosseriegrosplan_vindanslamoquette__port_506486', 'firstuploadexperveo_rouetranche__port_505677', 'firstuploadexperveo_vignette__port_505674', 'firstuploadexperveo_vin__port_505675', 'interieur_compteur_kilometrique__port_503644', 'uploadphotoimmatbest_class_1__port_505051']) select photo_hashtag_type from MTRDatou.classification_theme where id = 358 thcl : 358 photo_hashtag_type : 337 SELECT phi.hashtag_id , phi.photo_id FROM MTRBack.photo_hashtag_ids phi, MTRUser.mtr_portfolio_photos mp where phi.type = 337 and phi.photo_id = mp.mtr_photo_id and mp.mtr_portfolio_id =510365; {510365: [(917973295, 1), (917973297, 1), (917973302, 1), (917973293, 7), (917973296, 11), (917973300, 11), (917973286, 13), (917973289, 13), (917973301, 24), (917973285, 29), (917973290, 29), (917973299, 29), (917973304, 35), (917973287, 36), (917973298, 36), (917973305, 36), (917973292, 37), (917973291, 41), (917973303, 41), (917973294, 42), (917973288, 46)]} ############################### TEST rotate ################################ test rotate only Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=230 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=230 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 230 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=230 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.13268208503723145 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:rotate Thu Feb 27 00:40:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step_rotate ! We are in a linear step without datou_depend ! rotate photos of 90,180,270 degres batch 1 select photo_id, hashtag_id, `type`, x0, x1, y0, y1, score, id from MTRPhoto.crop_hashtag_ids where photo_id in ( 917849322) and `type` in (0) Loaded 0 chid ids of type : 0 SELECT crop_hashtag_id, points FROM MTRPhoto.crop_polygon_points WHERE crop_hashtag_id in () map_chi : {} photo_id in download_rotate_and_save : 917849322 list_chi_loc : 0 Use all angle ! Rotation of photo 917849322 of 90 degree temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 90 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] 90 [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] shrink_image : False image_rotate : image_path : temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 180 degree temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 180 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] 180 [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] shrink_image : False image_rotate : image_path : temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 270 degree temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 270 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] 270 [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] shrink_image : False image_rotate : image_path : temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg image_rotate.mode : RGB About to upload 3 photos upload in portfolio : 551782 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740613217_1084700 we have uploaded 3 photos in the portfolio 551782 time of upload the photos Elapsed time : 1.1615548133850098 map_filename_photo_id : 3 map_filename_photo_id : {'temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg': 1339823626, 'temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg': 1339823627, 'temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg': 1339823628} Len new_chis : 3 Len list_new_chi_with_photo_id : 0 of type : 0 list_new_chi_with_photo_id : [] After datou_step_exec type output : time spend for datou_step_exec : 1.3854353427886963 time spend to save output : 5.6743621826171875e-05 total time spend for step 1 : 1.3854920864105225 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : rotate we use saveGeneral [917849322] map_info['map_portfolio_photo'] : {} final : True mtd_id 230 list_pids : [917849322] Looping around the photos to save general results len do output : 3 /1339823626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339823627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339823628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('230', None, None, None, None, None, None, None, None) ('230', None, '917849322', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('230', None, '1339823626', 'None', None, None, None, None, None), ('230', None, '1339823627', 'None', None, None, None, None, None), ('230', None, '1339823628', 'None', None, None, None, None, None), ('230', None, '917849322', None, None, None, None, None, None)] time used for this insertion : 0.12807011604309082 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {1339823626: ['917849322', 'temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg', []], 1339823627: ['917849322', 'temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg', []], 1339823628: ['917849322', 'temp/1740613217_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg', []]} test rotate only is a success ! test rotate conditionnel Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=233 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=233 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 233 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=233 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax, rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.1472148895263672 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 3 step1:thcl Thu Feb 27 00:40:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740613219_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1740613219_1084700_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'500': 1} we are using the classfication for only one thcl 500 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (2448, 3264, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.00020384788513183594 time to convert the images to numpy array : 1.4928662776947021 total time to convert the images to numpy array : 1.493617296218872 list photo_ids error: [] list photo_ids correct : [917849322] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 500 To do loadFromThcl(), then load ParamDescType : thcl500 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (500) thcls : [{'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'}] thcl {'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'} Update svm_hashtag_type_desc : 3517 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) To loadFromThcl() : net_3517 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 28 wait 20 seconds l 3637 free memory gpu now : 28 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) param : , param.caffemodel : orientation_carte_grise_all_2 None mean_file_type : mean_file_path : prototxt_file_path : model : orientation_carte_grise_all_2 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : orientation_carte_grise_all_2 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/orientation_carte_grise_all_2 /data/models_weight/orientation_carte_grise_all_2/caffemodel size_local : 537110520 size in s3 : 537110520 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:17 caffemodel already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_fc.prototxt size_local : 1130 size in s3 : 1130 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt size_local : 5653 size in s3 : 5653 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:31 mean.npy already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/synset_words.txt size_local : 159 size in s3 : 159 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt caffemodel_filename : /data/models_weight/orientation_carte_grise_all_2/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 28 wait 20 seconds WARNING: Logging before InitGoogleLogging() is written to STDERR F0227 00:41:09.403985 1084700 syncedmem.cpp:78] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 39.59user 30.21system 5:44.76elapsed 20%CPU (0avgtext+0avgdata 4163688maxresident)k 6359720inputs+22664outputs (13599major+3495341minor)pagefaults 0swaps