python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3763 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.132615327835083 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu May 15 06:35:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3763 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-05-15 06:35:33.655790: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-15 06:35:33.695297: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-15 06:35:33.697646: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7a2c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-15 06:35:33.697741: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-15 06:35:33.702499: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-15 06:35:33.861507: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x242894c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-15 06:35:33.861578: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-15 06:35:33.863170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 06:35:33.863634: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:35:33.866628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:35:33.886508: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 06:35:33.887099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 06:35:33.919655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 06:35:33.924861: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 06:35:33.983576: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:35:33.984964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 06:35:33.985406: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:35:33.986105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-15 06:35:33.986125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-15 06:35:33.986136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-15 06:35:33.987705: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3311 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-05-15 06:35:34.736831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 06:35:34.736911: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:35:34.736928: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:35:34.736943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 06:35:34.736958: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 06:35:34.736972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 06:35:34.736999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 06:35:34.737015: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:35:34.737905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 06:35:34.738840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 06:35:34.738875: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:35:34.738891: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:35:34.738930: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 06:35:34.738952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 06:35:34.738973: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 06:35:34.738993: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 06:35:34.739009: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:35:34.739889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 06:35:34.739918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-15 06:35:34.739927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-15 06:35:34.739934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-15 06:35:34.740860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3311 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-05-15 06:35:43.683944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:35:44.030518: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:35:45.987531: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:45.988178: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.46G (2643237120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:45.988795: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.21G (2378913280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:45.988826: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:45.989519: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:45.989536: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:45.998864: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:45.998887: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:45.999490: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:45.999507: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:46.007324: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.007347: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:46.007926: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.007942: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:46.039008: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.039047: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:46.039646: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.039664: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:46.045484: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.045505: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:46.046082: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.046097: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-15 06:35:46.079578: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.080211: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.082075: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.082662: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.126386: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.127060: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.129272: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.129856: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.137655: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.138246: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.143107: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.143695: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.155968: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.156559: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.158246: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.158831: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.166682: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.167291: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.169167: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.169751: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.176830: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.177417: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.179140: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.179726: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.215793: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.216402: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.216974: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.217542: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.223505: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.224089: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.251883: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.252483: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.253054: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.253623: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.267619: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.268236: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.268810: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.269382: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.274093: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.274710: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.279524: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.280135: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.292425: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.293052: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.297285: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.297901: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.298476: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.299071: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.320961: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.321632: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.322232: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.322837: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.323430: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.324002: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.324571: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.325229: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.340595: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.341222: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.374797: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.374859: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-05-15 06:35:46.375857: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.376868: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.384119: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.384877: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.385878: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.386592: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.394981: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.395721: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.419034: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.420121: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.421178: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.422200: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.427808: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.428832: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.429810: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.430831: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.433449: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.445289: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.446271: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.458741: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.459776: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.460780: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.461744: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.462765: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-15 06:35:46.463825: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.73G (2936930304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (480, 640, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 1987014 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1771 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2964 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 2025-05-15 06:35:57.265377: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-15 06:35:57.291312: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-15 06:35:57.293668: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7a30000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-15 06:35:57.293749: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-15 06:35:57.319936: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-15 06:35:57.600673: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x24e6cf10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-15 06:35:57.600735: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-15 06:35:57.601789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 06:35:57.606166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:35:57.650857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:35:57.655057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 06:35:57.661166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 06:35:57.671951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 06:35:57.674311: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 06:35:57.683811: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:35:57.684999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 06:35:57.685129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:35:57.685777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-15 06:35:57.685794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-15 06:35:57.685803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-15 06:35:57.686603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 191 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-05-15 06:35:57.883661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 06:35:57.883796: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:35:57.883820: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:35:57.883842: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 06:35:57.883862: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 06:35:57.883883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 06:35:57.883903: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 06:35:57.883942: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:35:57.884750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 06:35:57.885859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 06:35:57.885901: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:35:57.885922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:35:57.885942: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 06:35:57.885962: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 06:35:57.885982: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 06:35:57.886001: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 06:35:57.886021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:35:57.886948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 06:35:57.887004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-15 06:35:57.887015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-15 06:35:57.887024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-15 06:35:57.887935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 191 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2025-05-15 06:36:10.084718: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 9.00MiB (rounded to 9437184) Current allocation summary follows. 2025-05-15 06:36:10.779639: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-05-15 06:36:10.779691: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 65, Chunks in use: 65. 16.2KiB allocated for chunks. 16.2KiB in use in bin. 8.9KiB client-requested in use in bin. 2025-05-15 06:36:10.779709: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-05-15 06:36:10.779726: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 251, Chunks in use: 251. 251.2KiB allocated for chunks. 251.2KiB in use in bin. 251.0KiB client-requested in use in bin. 2025-05-15 06:36:10.779742: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 40, Chunks in use: 40. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-05-15 06:36:10.779758: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 120, Chunks in use: 120. 487.5KiB allocated for chunks. 487.5KiB in use in bin. 480.0KiB client-requested in use in bin. 2025-05-15 06:36:10.779774: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 10, Chunks in use: 10. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-05-15 06:36:10.779789: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2025-05-15 06:36:10.779821: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-05-15 06:36:10.779838: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-05-15 06:36:10.779854: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-05-15 06:36:10.779869: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 8, Chunks in use: 7. 2.35MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2025-05-15 06:36:10.779884: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 7, Chunks in use: 6. 4.39MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2025-05-15 06:36:10.779900: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 48, Chunks in use: 45. 54.12MiB allocated for chunks. 50.62MiB in use in bin. 45.00MiB client-requested in use in bin. 2025-05-15 06:36:10.779916: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 28, Chunks in use: 25. 64.25MiB allocated for chunks. 55.75MiB in use in bin. 55.75MiB client-requested in use in bin. 2025-05-15 06:36:10.779931: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 4, Chunks in use: 2. 18.00MiB allocated for chunks. 8.00MiB in use in bin. 8.00MiB client-requested in use in bin. 2025-05-15 06:36:10.779946: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 5, Chunks in use: 4. 46.88MiB allocated for chunks. 38.88MiB in use in bin. 35.00MiB client-requested in use in bin. 2025-05-15 06:36:10.779961: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 06:36:10.779974: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 06:36:10.779988: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 06:36:10.780002: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 06:36:10.780015: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 06:36:10.780029: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 9.00MiB was 8.00MiB, Chunk State: 2025-05-15 06:36:10.780051: I tensorflow/core/common_runtime/bfc_allocator.cc:963] Size: 8.00MiB | Requested Size: 8.00MiB | in_use: 0 | bin_num: 15, prev: Size: 9.00MiB | Requested Size: 9.00MiB | in_use: 1 | bin_num: -1, next: Size: 8.00MiB | Requested Size: 8.00MiB | in_use: 1 | bin_num: -1 2025-05-15 06:36:10.780064: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 68026368 2025-05-15 06:36:10.780079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f784e000000 of size 4194304 next 632 2025-05-15 06:36:10.780093: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f784e400000 of size 4194304 next 617 2025-05-15 06:36:10.780114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f784e800000 of size 4194304 next 616 2025-05-15 06:36:10.780127: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f784ec00000 of size 6291456 next 608 2025-05-15 06:36:10.780139: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f784f200000 of size 9437184 next 607 2025-05-15 06:36:10.780151: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f784fb00000 of size 8388608 next 625 2025-05-15 06:36:10.780164: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7850300000 of size 8388608 next 624 2025-05-15 06:36:10.780176: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7850b00000 of size 9437184 next 637 2025-05-15 06:36:10.780188: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7851400000 of size 13500416 next 18446744073709551615 2025-05-15 06:36:10.780200: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 67108864 2025-05-15 06:36:10.780212: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7854000000 of size 2359296 next 370 2025-05-15 06:36:10.780225: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7854240000 of size 1048576 next 394 2025-05-15 06:36:10.780237: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7854340000 of size 1310720 next 388 2025-05-15 06:36:10.780249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7854480000 of size 2359296 next 387 2025-05-15 06:36:10.780261: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78546c0000 of size 1048576 next 412 2025-05-15 06:36:10.780274: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78547c0000 of size 1310720 next 406 2025-05-15 06:36:10.780286: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7854900000 of size 2359296 next 405 2025-05-15 06:36:10.780298: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7854b40000 of size 1048576 next 430 2025-05-15 06:36:10.780310: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7854c40000 of size 1310720 next 424 2025-05-15 06:36:10.780322: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7854d80000 of size 2359296 next 423 2025-05-15 06:36:10.780334: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7854fc0000 of size 1048576 next 447 2025-05-15 06:36:10.780346: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78550c0000 of size 1310720 next 441 2025-05-15 06:36:10.780358: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7855200000 of size 2359296 next 440 2025-05-15 06:36:10.780370: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7855440000 of size 1048576 next 465 2025-05-15 06:36:10.780382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7855540000 of size 1310720 next 459 2025-05-15 06:36:10.780394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7855680000 of size 2359296 next 458 2025-05-15 06:36:10.780406: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78558c0000 of size 1048576 next 483 2025-05-15 06:36:10.780418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78559c0000 of size 1310720 next 477 2025-05-15 06:36:10.780430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7855b00000 of size 2359296 next 476 2025-05-15 06:36:10.780442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7855d40000 of size 1048576 next 501 2025-05-15 06:36:10.780454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7855e40000 of size 1310720 next 495 2025-05-15 06:36:10.780466: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7855f80000 of size 2359296 next 494 2025-05-15 06:36:10.780478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78561c0000 of size 1048576 next 519 2025-05-15 06:36:10.780490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78562c0000 of size 1310720 next 513 2025-05-15 06:36:10.780510: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7856400000 of size 2359296 next 512 2025-05-15 06:36:10.780522: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7856640000 of size 1048576 next 537 2025-05-15 06:36:10.780534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7856740000 of size 1310720 next 531 2025-05-15 06:36:10.780546: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7856880000 of size 2359296 next 530 2025-05-15 06:36:10.780558: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7856ac0000 of size 1048576 next 555 2025-05-15 06:36:10.780570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7856bc0000 of size 1310720 next 549 2025-05-15 06:36:10.780582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7856d00000 of size 2359296 next 548 2025-05-15 06:36:10.780594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7856f40000 of size 1048576 next 573 2025-05-15 06:36:10.780606: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7857040000 of size 1310720 next 567 2025-05-15 06:36:10.780618: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7857180000 of size 2359296 next 566 2025-05-15 06:36:10.780630: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78573c0000 of size 1048576 next 591 2025-05-15 06:36:10.780642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f78574c0000 of size 1310720 next 585 2025-05-15 06:36:10.780654: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7857600000 of size 2359296 next 584 2025-05-15 06:36:10.780666: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7857840000 of size 2097152 next 597 2025-05-15 06:36:10.780679: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7857a40000 of size 2097152 next 596 2025-05-15 06:36:10.780691: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7857c40000 of size 3932160 next 18446744073709551615 2025-05-15 06:36:10.780703: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432 2025-05-15 06:36:10.780715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858000000 of size 2359296 next 245 2025-05-15 06:36:10.780727: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858240000 of size 1048576 next 269 2025-05-15 06:36:10.780739: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858340000 of size 1310720 next 263 2025-05-15 06:36:10.780751: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858480000 of size 2359296 next 262 2025-05-15 06:36:10.780763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78586c0000 of size 1048576 next 287 2025-05-15 06:36:10.780775: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78587c0000 of size 1310720 next 281 2025-05-15 06:36:10.780786: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858900000 of size 2359296 next 280 2025-05-15 06:36:10.780798: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858b40000 of size 1048576 next 304 2025-05-15 06:36:10.780811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858c40000 of size 1310720 next 298 2025-05-15 06:36:10.780823: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858d80000 of size 2359296 next 297 2025-05-15 06:36:10.780835: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7858fc0000 of size 1048576 next 322 2025-05-15 06:36:10.780847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78590c0000 of size 1310720 next 316 2025-05-15 06:36:10.780859: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7859200000 of size 2359296 next 315 2025-05-15 06:36:10.780870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7859440000 of size 1048576 next 340 2025-05-15 06:36:10.780890: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7859540000 of size 1310720 next 334 2025-05-15 06:36:10.780903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7859680000 of size 2359296 next 333 2025-05-15 06:36:10.780915: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78598c0000 of size 1048576 next 358 2025-05-15 06:36:10.780927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f78599c0000 of size 1310720 next 352 2025-05-15 06:36:10.780939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7859b00000 of size 2359296 next 351 2025-05-15 06:36:10.780951: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7859d40000 of size 1048576 next 376 2025-05-15 06:36:10.780963: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7859e40000 of size 1835008 next 18446744073709551615 2025-05-15 06:36:10.780975: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216 2025-05-15 06:36:10.780987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785a000000 of size 2359296 next 181 2025-05-15 06:36:10.780999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785a240000 of size 2097152 next 196 2025-05-15 06:36:10.781011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785a440000 of size 1048576 next 216 2025-05-15 06:36:10.781023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785a540000 of size 1310720 next 209 2025-05-15 06:36:10.781035: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785a680000 of size 2359296 next 208 2025-05-15 06:36:10.781047: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785a8c0000 of size 1048576 next 234 2025-05-15 06:36:10.781059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785a9c0000 of size 1310720 next 228 2025-05-15 06:36:10.781071: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785ab00000 of size 2359296 next 227 2025-05-15 06:36:10.781083: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f785ad40000 of size 2883584 next 18446744073709551615 2025-05-15 06:36:10.781094: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-05-15 06:36:10.781106: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c600000 of size 147456 next 55 2025-05-15 06:36:10.781119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c624000 of size 65536 next 78 2025-05-15 06:36:10.781131: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c634000 of size 4096 next 191 2025-05-15 06:36:10.781143: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c635000 of size 4096 next 192 2025-05-15 06:36:10.781155: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c636000 of size 4096 next 193 2025-05-15 06:36:10.781167: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c637000 of size 256 next 194 2025-05-15 06:36:10.781179: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c637100 of size 256 next 195 2025-05-15 06:36:10.781191: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c637200 of size 4096 next 197 2025-05-15 06:36:10.781203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c638200 of size 4096 next 198 2025-05-15 06:36:10.781215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c639200 of size 4096 next 199 2025-05-15 06:36:10.781227: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63a200 of size 4096 next 200 2025-05-15 06:36:10.781239: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63b200 of size 4096 next 201 2025-05-15 06:36:10.781251: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63c200 of size 1024 next 202 2025-05-15 06:36:10.781263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63c600 of size 1024 next 204 2025-05-15 06:36:10.781283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63ca00 of size 1024 next 205 2025-05-15 06:36:10.781295: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63ce00 of size 1024 next 206 2025-05-15 06:36:10.781307: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63d200 of size 1024 next 207 2025-05-15 06:36:10.781319: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63d600 of size 1024 next 210 2025-05-15 06:36:10.781331: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63da00 of size 1024 next 211 2025-05-15 06:36:10.781343: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63de00 of size 1024 next 212 2025-05-15 06:36:10.781354: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63e200 of size 1024 next 213 2025-05-15 06:36:10.781366: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63e600 of size 1024 next 214 2025-05-15 06:36:10.781378: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63ea00 of size 4096 next 215 2025-05-15 06:36:10.781390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c63fa00 of size 4096 next 217 2025-05-15 06:36:10.781402: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c640a00 of size 4096 next 218 2025-05-15 06:36:10.781414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c641a00 of size 4096 next 219 2025-05-15 06:36:10.781426: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c642a00 of size 4096 next 220 2025-05-15 06:36:10.781438: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c643a00 of size 1024 next 221 2025-05-15 06:36:10.781450: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c643e00 of size 1024 next 223 2025-05-15 06:36:10.781461: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c644200 of size 1024 next 224 2025-05-15 06:36:10.781473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c644600 of size 1024 next 225 2025-05-15 06:36:10.781485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c644a00 of size 1024 next 226 2025-05-15 06:36:10.781497: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c644e00 of size 1024 next 229 2025-05-15 06:36:10.781509: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c645200 of size 1024 next 230 2025-05-15 06:36:10.781521: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c645600 of size 1024 next 231 2025-05-15 06:36:10.781533: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c645a00 of size 1024 next 232 2025-05-15 06:36:10.781545: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c645e00 of size 1024 next 233 2025-05-15 06:36:10.781557: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c646200 of size 7680 next 72 2025-05-15 06:36:10.781569: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c648000 of size 147456 next 71 2025-05-15 06:36:10.781581: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c66c000 of size 131072 next 87 2025-05-15 06:36:10.781593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c68c000 of size 4096 next 289 2025-05-15 06:36:10.781605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c68d000 of size 4096 next 290 2025-05-15 06:36:10.781617: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c68e000 of size 4096 next 291 2025-05-15 06:36:10.781629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c68f000 of size 1024 next 292 2025-05-15 06:36:10.781641: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c68f400 of size 1024 next 293 2025-05-15 06:36:10.781653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c68f800 of size 1024 next 294 2025-05-15 06:36:10.781673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c68fc00 of size 1024 next 295 2025-05-15 06:36:10.781685: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c690000 of size 1024 next 296 2025-05-15 06:36:10.781697: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c690400 of size 1024 next 299 2025-05-15 06:36:10.781709: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c690800 of size 1024 next 300 2025-05-15 06:36:10.781721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c690c00 of size 1024 next 301 2025-05-15 06:36:10.781733: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c691000 of size 1024 next 302 2025-05-15 06:36:10.781744: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c691400 of size 1024 next 303 2025-05-15 06:36:10.781756: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c691800 of size 4096 next 305 2025-05-15 06:36:10.781768: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c692800 of size 4096 next 306 2025-05-15 06:36:10.781780: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c693800 of size 4096 next 307 2025-05-15 06:36:10.781792: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c694800 of size 4096 next 308 2025-05-15 06:36:10.781804: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c695800 of size 4096 next 309 2025-05-15 06:36:10.781816: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c696800 of size 1024 next 310 2025-05-15 06:36:10.781828: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c696c00 of size 1024 next 311 2025-05-15 06:36:10.781840: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c697000 of size 1024 next 312 2025-05-15 06:36:10.781852: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c697400 of size 1024 next 313 2025-05-15 06:36:10.781864: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c697800 of size 1024 next 314 2025-05-15 06:36:10.781876: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c697c00 of size 1024 next 317 2025-05-15 06:36:10.781888: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c698000 of size 1024 next 318 2025-05-15 06:36:10.781899: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c698400 of size 1024 next 319 2025-05-15 06:36:10.781911: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c698800 of size 1024 next 320 2025-05-15 06:36:10.781923: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c698c00 of size 1024 next 321 2025-05-15 06:36:10.781935: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c699000 of size 4096 next 323 2025-05-15 06:36:10.781947: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69a000 of size 4096 next 324 2025-05-15 06:36:10.781959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69b000 of size 4096 next 325 2025-05-15 06:36:10.781971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69c000 of size 4096 next 326 2025-05-15 06:36:10.781983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69d000 of size 4096 next 327 2025-05-15 06:36:10.781995: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69e000 of size 1024 next 328 2025-05-15 06:36:10.782006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69e400 of size 1024 next 329 2025-05-15 06:36:10.782018: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69e800 of size 1024 next 330 2025-05-15 06:36:10.782030: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69ec00 of size 1024 next 331 2025-05-15 06:36:10.782042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69f000 of size 1024 next 332 2025-05-15 06:36:10.782054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69f400 of size 1024 next 335 2025-05-15 06:36:10.782074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69f800 of size 1024 next 336 2025-05-15 06:36:10.782086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c69fc00 of size 1024 next 337 2025-05-15 06:36:10.782098: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a0000 of size 1024 next 338 2025-05-15 06:36:10.782110: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a0400 of size 1024 next 339 2025-05-15 06:36:10.782122: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a0800 of size 4096 next 341 2025-05-15 06:36:10.782134: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a1800 of size 4096 next 342 2025-05-15 06:36:10.782146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a2800 of size 4096 next 343 2025-05-15 06:36:10.782158: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a3800 of size 4096 next 344 2025-05-15 06:36:10.782170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a4800 of size 4096 next 345 2025-05-15 06:36:10.782181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a5800 of size 1024 next 346 2025-05-15 06:36:10.782193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a5c00 of size 1024 next 347 2025-05-15 06:36:10.782205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a6000 of size 1024 next 348 2025-05-15 06:36:10.782217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a6400 of size 1024 next 349 2025-05-15 06:36:10.782229: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a6800 of size 1024 next 350 2025-05-15 06:36:10.782241: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a6c00 of size 1024 next 353 2025-05-15 06:36:10.782253: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a7000 of size 1024 next 354 2025-05-15 06:36:10.782265: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a7400 of size 1024 next 355 2025-05-15 06:36:10.782277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a7800 of size 1024 next 356 2025-05-15 06:36:10.782289: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a7c00 of size 1024 next 357 2025-05-15 06:36:10.782301: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a8000 of size 4096 next 359 2025-05-15 06:36:10.782313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6a9000 of size 4096 next 360 2025-05-15 06:36:10.782325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6aa000 of size 4096 next 361 2025-05-15 06:36:10.782337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ab000 of size 4096 next 362 2025-05-15 06:36:10.782349: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ac000 of size 4096 next 363 2025-05-15 06:36:10.782360: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ad000 of size 1024 next 364 2025-05-15 06:36:10.782372: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ad400 of size 1024 next 365 2025-05-15 06:36:10.782384: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ad800 of size 1024 next 366 2025-05-15 06:36:10.782396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6adc00 of size 1024 next 367 2025-05-15 06:36:10.782408: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ae000 of size 1024 next 368 2025-05-15 06:36:10.782420: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ae400 of size 1024 next 371 2025-05-15 06:36:10.782432: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ae800 of size 1024 next 372 2025-05-15 06:36:10.782444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6aec00 of size 1024 next 373 2025-05-15 06:36:10.782464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6af000 of size 1024 next 374 2025-05-15 06:36:10.782476: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6af400 of size 1024 next 375 2025-05-15 06:36:10.782488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6af800 of size 4096 next 377 2025-05-15 06:36:10.782500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b0800 of size 4096 next 378 2025-05-15 06:36:10.782512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b1800 of size 4096 next 379 2025-05-15 06:36:10.782524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b2800 of size 4096 next 380 2025-05-15 06:36:10.782536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b3800 of size 4096 next 381 2025-05-15 06:36:10.782547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b4800 of size 1024 next 382 2025-05-15 06:36:10.782559: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b4c00 of size 1024 next 383 2025-05-15 06:36:10.782571: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b5000 of size 1024 next 384 2025-05-15 06:36:10.782583: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b5400 of size 1024 next 385 2025-05-15 06:36:10.782595: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b5800 of size 1024 next 386 2025-05-15 06:36:10.782607: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b5c00 of size 1024 next 389 2025-05-15 06:36:10.782619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b6000 of size 1024 next 390 2025-05-15 06:36:10.782631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b6400 of size 1024 next 391 2025-05-15 06:36:10.782643: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b6800 of size 1024 next 392 2025-05-15 06:36:10.782655: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b6c00 of size 1024 next 393 2025-05-15 06:36:10.782666: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b7000 of size 4096 next 395 2025-05-15 06:36:10.782678: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b8000 of size 4096 next 396 2025-05-15 06:36:10.782690: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6b9000 of size 4096 next 397 2025-05-15 06:36:10.782702: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ba000 of size 4096 next 398 2025-05-15 06:36:10.782714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bb000 of size 4096 next 399 2025-05-15 06:36:10.782726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bc000 of size 1024 next 400 2025-05-15 06:36:10.782738: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bc400 of size 1024 next 401 2025-05-15 06:36:10.782750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bc800 of size 1024 next 402 2025-05-15 06:36:10.782762: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bcc00 of size 1024 next 403 2025-05-15 06:36:10.782774: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bd000 of size 1024 next 404 2025-05-15 06:36:10.782786: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bd400 of size 1024 next 407 2025-05-15 06:36:10.782798: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bd800 of size 1024 next 408 2025-05-15 06:36:10.782810: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bdc00 of size 1024 next 409 2025-05-15 06:36:10.782822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6be000 of size 1024 next 410 2025-05-15 06:36:10.782833: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6be400 of size 1024 next 411 2025-05-15 06:36:10.782853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6be800 of size 4096 next 413 2025-05-15 06:36:10.782865: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6bf800 of size 4096 next 414 2025-05-15 06:36:10.782877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c0800 of size 4096 next 415 2025-05-15 06:36:10.782889: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c1800 of size 4096 next 416 2025-05-15 06:36:10.782911: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c2800 of size 4096 next 417 2025-05-15 06:36:10.782926: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c3800 of size 1024 next 418 2025-05-15 06:36:10.782938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c3c00 of size 1024 next 419 2025-05-15 06:36:10.782950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c4000 of size 1024 next 420 2025-05-15 06:36:10.782962: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c4400 of size 1024 next 421 2025-05-15 06:36:10.782974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c4800 of size 1024 next 422 2025-05-15 06:36:10.782986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c4c00 of size 1024 next 425 2025-05-15 06:36:10.782998: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c5000 of size 1024 next 426 2025-05-15 06:36:10.783009: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c5400 of size 1024 next 427 2025-05-15 06:36:10.783021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c5800 of size 1024 next 428 2025-05-15 06:36:10.783033: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c5c00 of size 1024 next 429 2025-05-15 06:36:10.783045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c6000 of size 4096 next 431 2025-05-15 06:36:10.783057: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c7000 of size 4096 next 432 2025-05-15 06:36:10.783069: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c8000 of size 4096 next 433 2025-05-15 06:36:10.783081: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6c9000 of size 4096 next 434 2025-05-15 06:36:10.783093: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6ca000 of size 4096 next 435 2025-05-15 06:36:10.783104: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6cb000 of size 1024 next 436 2025-05-15 06:36:10.783116: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6cb400 of size 1024 next 437 2025-05-15 06:36:10.783128: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6cb800 of size 1024 next 438 2025-05-15 06:36:10.783140: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6cbc00 of size 1024 next 103 2025-05-15 06:36:10.783152: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c6cc000 of size 262144 next 102 2025-05-15 06:36:10.783165: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f785c70c000 of size 999424 next 18446744073709551615 2025-05-15 06:36:10.783176: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-05-15 06:36:10.783189: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c800000 of size 589824 next 96 2025-05-15 06:36:10.783201: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c890000 of size 262144 next 117 2025-05-15 06:36:10.783213: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d0000 of size 1024 next 439 2025-05-15 06:36:10.783225: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d0400 of size 1024 next 442 2025-05-15 06:36:10.783237: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d0800 of size 1024 next 443 2025-05-15 06:36:10.783257: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d0c00 of size 1024 next 444 2025-05-15 06:36:10.783270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d1000 of size 1024 next 445 2025-05-15 06:36:10.783282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d1400 of size 1024 next 446 2025-05-15 06:36:10.783293: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d1800 of size 4096 next 448 2025-05-15 06:36:10.783305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d2800 of size 4096 next 449 2025-05-15 06:36:10.783317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d3800 of size 4096 next 450 2025-05-15 06:36:10.783329: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d4800 of size 4096 next 451 2025-05-15 06:36:10.783341: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d5800 of size 4096 next 452 2025-05-15 06:36:10.783353: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d6800 of size 1024 next 453 2025-05-15 06:36:10.783365: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d6c00 of size 1024 next 454 2025-05-15 06:36:10.783377: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d7000 of size 1024 next 455 2025-05-15 06:36:10.783389: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d7400 of size 1024 next 456 2025-05-15 06:36:10.783401: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d7800 of size 1024 next 457 2025-05-15 06:36:10.783413: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d7c00 of size 1024 next 460 2025-05-15 06:36:10.783424: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d8000 of size 1024 next 461 2025-05-15 06:36:10.783436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d8400 of size 1024 next 462 2025-05-15 06:36:10.783448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d8800 of size 1024 next 463 2025-05-15 06:36:10.783460: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d8c00 of size 1024 next 464 2025-05-15 06:36:10.783472: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8d9000 of size 4096 next 466 2025-05-15 06:36:10.783484: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8da000 of size 4096 next 467 2025-05-15 06:36:10.783496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8db000 of size 4096 next 468 2025-05-15 06:36:10.783508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8dc000 of size 4096 next 469 2025-05-15 06:36:10.783520: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8dd000 of size 4096 next 470 2025-05-15 06:36:10.783532: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8de000 of size 1024 next 471 2025-05-15 06:36:10.783544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8de400 of size 1024 next 472 2025-05-15 06:36:10.783555: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8de800 of size 1024 next 473 2025-05-15 06:36:10.783567: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8dec00 of size 1024 next 474 2025-05-15 06:36:10.783579: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8df000 of size 1024 next 475 2025-05-15 06:36:10.783591: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8df400 of size 1024 next 478 2025-05-15 06:36:10.783603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8df800 of size 1024 next 479 2025-05-15 06:36:10.783615: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8dfc00 of size 1024 next 480 2025-05-15 06:36:10.783627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e0000 of size 1024 next 481 2025-05-15 06:36:10.783638: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e0400 of size 1024 next 482 2025-05-15 06:36:10.783658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e0800 of size 4096 next 484 2025-05-15 06:36:10.783671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e1800 of size 4096 next 485 2025-05-15 06:36:10.783683: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e2800 of size 4096 next 486 2025-05-15 06:36:10.783695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e3800 of size 4096 next 487 2025-05-15 06:36:10.783706: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e4800 of size 4096 next 488 2025-05-15 06:36:10.783718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e5800 of size 1024 next 489 2025-05-15 06:36:10.783730: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e5c00 of size 1024 next 490 2025-05-15 06:36:10.783742: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e6000 of size 1024 next 491 2025-05-15 06:36:10.783754: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e6400 of size 1024 next 492 2025-05-15 06:36:10.783766: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e6800 of size 1024 next 493 2025-05-15 06:36:10.783778: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e6c00 of size 1024 next 496 2025-05-15 06:36:10.783790: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e7000 of size 1024 next 497 2025-05-15 06:36:10.783801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e7400 of size 1024 next 498 2025-05-15 06:36:10.783813: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e7800 of size 1024 next 499 2025-05-15 06:36:10.783825: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e7c00 of size 1024 next 500 2025-05-15 06:36:10.783837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e8000 of size 4096 next 502 2025-05-15 06:36:10.783849: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8e9000 of size 4096 next 503 2025-05-15 06:36:10.783861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ea000 of size 4096 next 504 2025-05-15 06:36:10.783873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8eb000 of size 4096 next 505 2025-05-15 06:36:10.783885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ec000 of size 4096 next 506 2025-05-15 06:36:10.783896: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ed000 of size 1024 next 507 2025-05-15 06:36:10.783908: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ed400 of size 1024 next 508 2025-05-15 06:36:10.783920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ed800 of size 1024 next 509 2025-05-15 06:36:10.783932: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8edc00 of size 1024 next 510 2025-05-15 06:36:10.783944: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ee000 of size 1024 next 511 2025-05-15 06:36:10.783956: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ee400 of size 1024 next 514 2025-05-15 06:36:10.783968: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ee800 of size 1024 next 515 2025-05-15 06:36:10.783979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8eec00 of size 1024 next 516 2025-05-15 06:36:10.783991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ef000 of size 1024 next 517 2025-05-15 06:36:10.784003: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ef400 of size 1024 next 518 2025-05-15 06:36:10.784015: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ef800 of size 4096 next 520 2025-05-15 06:36:10.784027: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f0800 of size 4096 next 521 2025-05-15 06:36:10.784047: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f1800 of size 4096 next 522 2025-05-15 06:36:10.784059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f2800 of size 4096 next 523 2025-05-15 06:36:10.784071: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f3800 of size 4096 next 524 2025-05-15 06:36:10.784083: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f4800 of size 1024 next 525 2025-05-15 06:36:10.784095: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f4c00 of size 1024 next 526 2025-05-15 06:36:10.784107: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f5000 of size 1024 next 527 2025-05-15 06:36:10.784119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f5400 of size 1024 next 528 2025-05-15 06:36:10.784130: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f5800 of size 1024 next 529 2025-05-15 06:36:10.784142: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f5c00 of size 1024 next 532 2025-05-15 06:36:10.784154: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f6000 of size 1024 next 533 2025-05-15 06:36:10.784166: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f6400 of size 1024 next 534 2025-05-15 06:36:10.784178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f6800 of size 1024 next 535 2025-05-15 06:36:10.784190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f6c00 of size 1024 next 536 2025-05-15 06:36:10.784202: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f7000 of size 4096 next 538 2025-05-15 06:36:10.784214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f8000 of size 4096 next 539 2025-05-15 06:36:10.784226: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8f9000 of size 4096 next 540 2025-05-15 06:36:10.784238: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fa000 of size 4096 next 541 2025-05-15 06:36:10.784250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fb000 of size 4096 next 542 2025-05-15 06:36:10.784262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fc000 of size 1024 next 543 2025-05-15 06:36:10.784274: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fc400 of size 1024 next 544 2025-05-15 06:36:10.784285: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fc800 of size 1024 next 545 2025-05-15 06:36:10.784297: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fcc00 of size 1024 next 546 2025-05-15 06:36:10.784309: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fd000 of size 1024 next 547 2025-05-15 06:36:10.784321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fd400 of size 1024 next 550 2025-05-15 06:36:10.784333: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fd800 of size 1024 next 551 2025-05-15 06:36:10.784345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fdc00 of size 1024 next 552 2025-05-15 06:36:10.784357: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fe000 of size 1024 next 553 2025-05-15 06:36:10.784369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fe400 of size 1024 next 554 2025-05-15 06:36:10.784381: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8fe800 of size 4096 next 556 2025-05-15 06:36:10.784393: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c8ff800 of size 4096 next 557 2025-05-15 06:36:10.784404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c900800 of size 4096 next 558 2025-05-15 06:36:10.784416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c901800 of size 4096 next 559 2025-05-15 06:36:10.784436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c902800 of size 4096 next 560 2025-05-15 06:36:10.784449: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c903800 of size 1024 next 561 2025-05-15 06:36:10.784461: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c903c00 of size 1024 next 562 2025-05-15 06:36:10.784473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c904000 of size 1024 next 563 2025-05-15 06:36:10.784484: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c904400 of size 1024 next 564 2025-05-15 06:36:10.784496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c904800 of size 1024 next 565 2025-05-15 06:36:10.784508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c904c00 of size 1024 next 568 2025-05-15 06:36:10.784520: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c905000 of size 1024 next 569 2025-05-15 06:36:10.784532: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c905400 of size 1024 next 570 2025-05-15 06:36:10.784544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c905800 of size 1024 next 571 2025-05-15 06:36:10.784556: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c905c00 of size 1024 next 572 2025-05-15 06:36:10.784568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c906000 of size 4096 next 574 2025-05-15 06:36:10.784579: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c907000 of size 4096 next 575 2025-05-15 06:36:10.784591: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c908000 of size 4096 next 576 2025-05-15 06:36:10.784603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c909000 of size 4096 next 577 2025-05-15 06:36:10.784615: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90a000 of size 4096 next 578 2025-05-15 06:36:10.784627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90b000 of size 1024 next 579 2025-05-15 06:36:10.784639: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90b400 of size 1024 next 580 2025-05-15 06:36:10.784651: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90b800 of size 1024 next 581 2025-05-15 06:36:10.784663: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90bc00 of size 1024 next 582 2025-05-15 06:36:10.784675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90c000 of size 1024 next 583 2025-05-15 06:36:10.784686: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90c400 of size 1024 next 586 2025-05-15 06:36:10.784698: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90c800 of size 1024 next 587 2025-05-15 06:36:10.784710: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90cc00 of size 1024 next 588 2025-05-15 06:36:10.784722: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90d000 of size 1024 next 589 2025-05-15 06:36:10.784734: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90d400 of size 1024 next 590 2025-05-15 06:36:10.784746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90d800 of size 4096 next 592 2025-05-15 06:36:10.784758: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c90e800 of size 6144 next 112 2025-05-15 06:36:10.784770: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c910000 of size 524288 next 111 2025-05-15 06:36:10.784782: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c990000 of size 262144 next 135 2025-05-15 06:36:10.784795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785c9d0000 of size 327680 next 124 2025-05-15 06:36:10.784807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785ca20000 of size 589824 next 123 2025-05-15 06:36:10.784826: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cab0000 of size 262144 next 152 2025-05-15 06:36:10.784839: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785caf0000 of size 327680 next 142 2025-05-15 06:36:10.784851: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cb40000 of size 786432 next 18446744073709551615 2025-05-15 06:36:10.784863: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-05-15 06:36:10.784875: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc00000 of size 4096 next 593 2025-05-15 06:36:10.784887: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc01000 of size 4096 next 594 2025-05-15 06:36:10.784899: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc02000 of size 4096 next 595 2025-05-15 06:36:10.784911: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc03000 of size 2048 next 598 2025-05-15 06:36:10.784923: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc03800 of size 2048 next 599 2025-05-15 06:36:10.784935: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc04000 of size 2048 next 600 2025-05-15 06:36:10.784947: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc04800 of size 2048 next 601 2025-05-15 06:36:10.784959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc05000 of size 2048 next 602 2025-05-15 06:36:10.784971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc05800 of size 256 next 605 2025-05-15 06:36:10.784983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc05900 of size 256 next 606 2025-05-15 06:36:10.784994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc05a00 of size 2048 next 604 2025-05-15 06:36:10.785006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc06200 of size 2048 next 609 2025-05-15 06:36:10.785018: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc06a00 of size 2048 next 610 2025-05-15 06:36:10.785030: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc07200 of size 2048 next 611 2025-05-15 06:36:10.785042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc07a00 of size 2048 next 612 2025-05-15 06:36:10.785054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc08200 of size 256 next 614 2025-05-15 06:36:10.785066: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc08300 of size 256 next 615 2025-05-15 06:36:10.785078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc08400 of size 8192 next 613 2025-05-15 06:36:10.785090: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc0a400 of size 8192 next 618 2025-05-15 06:36:10.785102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc0c400 of size 8192 next 619 2025-05-15 06:36:10.785114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc0e400 of size 8192 next 620 2025-05-15 06:36:10.785126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc10400 of size 8192 next 621 2025-05-15 06:36:10.785138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc12400 of size 256 next 622 2025-05-15 06:36:10.785150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc12500 of size 256 next 623 2025-05-15 06:36:10.785162: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc12600 of size 8192 next 626 2025-05-15 06:36:10.785174: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc14600 of size 8192 next 627 2025-05-15 06:36:10.785186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc16600 of size 8192 next 628 2025-05-15 06:36:10.785198: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc18600 of size 8192 next 629 2025-05-15 06:36:10.785210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc1a600 of size 8192 next 630 2025-05-15 06:36:10.785233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc1c600 of size 2048 next 631 2025-05-15 06:36:10.785246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc1ce00 of size 2048 next 633 2025-05-15 06:36:10.785258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc1d600 of size 2048 next 634 2025-05-15 06:36:10.785270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc1de00 of size 2048 next 635 2025-05-15 06:36:10.785282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc1e600 of size 2048 next 636 2025-05-15 06:36:10.785294: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f785cc1ee00 of size 463360 next 160 2025-05-15 06:36:10.785306: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cc90000 of size 589824 next 158 2025-05-15 06:36:10.785319: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cd20000 of size 524288 next 171 2025-05-15 06:36:10.785331: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f785cda0000 of size 1048576 next 222 2025-05-15 06:36:10.785343: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cea0000 of size 1048576 next 190 2025-05-15 06:36:10.785355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785cfa0000 of size 1048576 next 189 2025-05-15 06:36:10.785367: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785d0a0000 of size 1048576 next 203 2025-05-15 06:36:10.785379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785d1a0000 of size 1048576 next 251 2025-05-15 06:36:10.785392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f785d2a0000 of size 1441792 next 18446744073709551615 2025-05-15 06:36:10.785403: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-05-15 06:36:10.785416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00000 of size 1280 next 1 2025-05-15 06:36:10.785428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00500 of size 256 next 5 2025-05-15 06:36:10.785440: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00600 of size 256 next 8 2025-05-15 06:36:10.785452: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00700 of size 256 next 9 2025-05-15 06:36:10.785464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00800 of size 256 next 10 2025-05-15 06:36:10.785476: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00900 of size 256 next 11 2025-05-15 06:36:10.785488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00a00 of size 256 next 12 2025-05-15 06:36:10.785500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00b00 of size 256 next 13 2025-05-15 06:36:10.785511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00c00 of size 256 next 17 2025-05-15 06:36:10.785524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00d00 of size 256 next 19 2025-05-15 06:36:10.785536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00e00 of size 256 next 20 2025-05-15 06:36:10.785548: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e00f00 of size 256 next 21 2025-05-15 06:36:10.785560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01000 of size 256 next 22 2025-05-15 06:36:10.785572: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01100 of size 256 next 24 2025-05-15 06:36:10.785584: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01200 of size 256 next 25 2025-05-15 06:36:10.785595: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01300 of size 256 next 23 2025-05-15 06:36:10.785607: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01400 of size 256 next 28 2025-05-15 06:36:10.785627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01500 of size 256 next 29 2025-05-15 06:36:10.785640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01600 of size 256 next 30 2025-05-15 06:36:10.785652: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01700 of size 256 next 31 2025-05-15 06:36:10.785664: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01800 of size 256 next 33 2025-05-15 06:36:10.785676: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01900 of size 256 next 34 2025-05-15 06:36:10.785688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01a00 of size 1024 next 32 2025-05-15 06:36:10.785700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e01e00 of size 1024 next 37 2025-05-15 06:36:10.785712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e02200 of size 1024 next 38 2025-05-15 06:36:10.785724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e02600 of size 1024 next 39 2025-05-15 06:36:10.785736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e02a00 of size 1024 next 40 2025-05-15 06:36:10.785748: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e02e00 of size 1024 next 41 2025-05-15 06:36:10.785760: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e03200 of size 1024 next 43 2025-05-15 06:36:10.785772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e03600 of size 1024 next 44 2025-05-15 06:36:10.785784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e03a00 of size 1024 next 45 2025-05-15 06:36:10.785796: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e03e00 of size 1024 next 46 2025-05-15 06:36:10.785808: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04200 of size 256 next 48 2025-05-15 06:36:10.785820: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04300 of size 256 next 49 2025-05-15 06:36:10.785831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04400 of size 256 next 50 2025-05-15 06:36:10.785843: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04500 of size 256 next 51 2025-05-15 06:36:10.785855: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04600 of size 256 next 52 2025-05-15 06:36:10.785867: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04700 of size 256 next 53 2025-05-15 06:36:10.785880: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04800 of size 256 next 56 2025-05-15 06:36:10.785892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04900 of size 256 next 57 2025-05-15 06:36:10.785904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04a00 of size 256 next 58 2025-05-15 06:36:10.785916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04b00 of size 256 next 14 2025-05-15 06:36:10.785928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04c00 of size 256 next 15 2025-05-15 06:36:10.785940: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04d00 of size 256 next 16 2025-05-15 06:36:10.785952: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e04e00 of size 1024 next 60 2025-05-15 06:36:10.785964: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e05200 of size 1024 next 61 2025-05-15 06:36:10.785976: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e05600 of size 1024 next 62 2025-05-15 06:36:10.785987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e05a00 of size 1024 next 63 2025-05-15 06:36:10.785999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e05e00 of size 1024 next 64 2025-05-15 06:36:10.786011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06200 of size 256 next 66 2025-05-15 06:36:10.786031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06300 of size 256 next 67 2025-05-15 06:36:10.786043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06400 of size 256 next 68 2025-05-15 06:36:10.786055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06500 of size 256 next 69 2025-05-15 06:36:10.786067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06600 of size 256 next 70 2025-05-15 06:36:10.786079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06700 of size 256 next 73 2025-05-15 06:36:10.786091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06800 of size 256 next 74 2025-05-15 06:36:10.786103: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06900 of size 256 next 75 2025-05-15 06:36:10.786115: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06a00 of size 256 next 76 2025-05-15 06:36:10.786126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06b00 of size 256 next 77 2025-05-15 06:36:10.786138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e06c00 of size 1024 next 79 2025-05-15 06:36:10.786150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e07000 of size 1024 next 80 2025-05-15 06:36:10.786162: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e07400 of size 1024 next 81 2025-05-15 06:36:10.786174: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e07800 of size 1024 next 82 2025-05-15 06:36:10.786186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e07c00 of size 1024 next 83 2025-05-15 06:36:10.786198: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08000 of size 256 next 85 2025-05-15 06:36:10.786210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08100 of size 256 next 86 2025-05-15 06:36:10.786222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08200 of size 512 next 84 2025-05-15 06:36:10.786234: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08400 of size 512 next 88 2025-05-15 06:36:10.786246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08600 of size 512 next 89 2025-05-15 06:36:10.786258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08800 of size 512 next 90 2025-05-15 06:36:10.786270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08a00 of size 512 next 91 2025-05-15 06:36:10.786282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08c00 of size 256 next 93 2025-05-15 06:36:10.786294: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08d00 of size 256 next 94 2025-05-15 06:36:10.786305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e08e00 of size 512 next 92 2025-05-15 06:36:10.786317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e09000 of size 512 next 97 2025-05-15 06:36:10.786329: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e09200 of size 512 next 98 2025-05-15 06:36:10.786341: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e09400 of size 512 next 99 2025-05-15 06:36:10.786353: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e09600 of size 512 next 2 2025-05-15 06:36:10.786365: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e09800 of size 256 next 3 2025-05-15 06:36:10.786377: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e09900 of size 256 next 4 2025-05-15 06:36:10.786389: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e09a00 of size 16384 next 18 2025-05-15 06:36:10.786401: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e0da00 of size 256 next 100 2025-05-15 06:36:10.786422: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e0db00 of size 256 next 101 2025-05-15 06:36:10.786434: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e0dc00 of size 2048 next 104 2025-05-15 06:36:10.786446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e0e400 of size 2048 next 105 2025-05-15 06:36:10.786458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e0ec00 of size 2048 next 106 2025-05-15 06:36:10.786470: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e0f400 of size 2048 next 107 2025-05-15 06:36:10.786482: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e0fc00 of size 2048 next 108 2025-05-15 06:36:10.786494: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e10400 of size 256 next 109 2025-05-15 06:36:10.786506: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e10500 of size 256 next 110 2025-05-15 06:36:10.786518: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e10600 of size 2048 next 113 2025-05-15 06:36:10.786530: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e10e00 of size 2048 next 114 2025-05-15 06:36:10.786542: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e11600 of size 2048 next 115 2025-05-15 06:36:10.786553: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e11e00 of size 2048 next 116 2025-05-15 06:36:10.786565: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e12600 of size 2048 next 6 2025-05-15 06:36:10.786578: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e12e00 of size 37632 next 7 2025-05-15 06:36:10.786590: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1c100 of size 512 next 118 2025-05-15 06:36:10.786601: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1c300 of size 512 next 119 2025-05-15 06:36:10.786613: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1c500 of size 512 next 120 2025-05-15 06:36:10.786625: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1c700 of size 512 next 121 2025-05-15 06:36:10.786637: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1c900 of size 512 next 122 2025-05-15 06:36:10.786649: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1cb00 of size 512 next 125 2025-05-15 06:36:10.786661: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1cd00 of size 512 next 126 2025-05-15 06:36:10.786673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1cf00 of size 512 next 127 2025-05-15 06:36:10.786685: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1d100 of size 512 next 128 2025-05-15 06:36:10.786696: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1d300 of size 512 next 129 2025-05-15 06:36:10.786708: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1d500 of size 2048 next 130 2025-05-15 06:36:10.786720: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1dd00 of size 2048 next 131 2025-05-15 06:36:10.786732: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1e500 of size 2048 next 132 2025-05-15 06:36:10.786744: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1ed00 of size 2048 next 133 2025-05-15 06:36:10.786756: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1f500 of size 2048 next 134 2025-05-15 06:36:10.786768: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1fd00 of size 512 next 136 2025-05-15 06:36:10.786780: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e1ff00 of size 512 next 137 2025-05-15 06:36:10.786792: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e20100 of size 512 next 138 2025-05-15 06:36:10.786803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e20300 of size 512 next 139 2025-05-15 06:36:10.786832: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e20500 of size 512 next 140 2025-05-15 06:36:10.786844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e20700 of size 512 next 141 2025-05-15 06:36:10.786856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e20900 of size 512 next 143 2025-05-15 06:36:10.786868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e20b00 of size 512 next 144 2025-05-15 06:36:10.786880: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e20d00 of size 512 next 145 2025-05-15 06:36:10.786892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e20f00 of size 512 next 146 2025-05-15 06:36:10.786950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e21100 of size 2048 next 147 2025-05-15 06:36:10.786963: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e21900 of size 2048 next 148 2025-05-15 06:36:10.786975: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e22100 of size 2048 next 149 2025-05-15 06:36:10.786987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e22900 of size 2048 next 150 2025-05-15 06:36:10.786998: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e23100 of size 2048 next 151 2025-05-15 06:36:10.787010: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e23900 of size 512 next 153 2025-05-15 06:36:10.787022: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e23b00 of size 512 next 154 2025-05-15 06:36:10.787034: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e23d00 of size 512 next 155 2025-05-15 06:36:10.787049: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e23f00 of size 512 next 156 2025-05-15 06:36:10.787061: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e24100 of size 512 next 157 2025-05-15 06:36:10.787073: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e24300 of size 512 next 161 2025-05-15 06:36:10.787084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e24500 of size 512 next 162 2025-05-15 06:36:10.787096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e24700 of size 512 next 163 2025-05-15 06:36:10.787108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e24900 of size 512 next 164 2025-05-15 06:36:10.787120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e24b00 of size 512 next 165 2025-05-15 06:36:10.787132: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e24d00 of size 2048 next 166 2025-05-15 06:36:10.787144: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e25500 of size 2048 next 167 2025-05-15 06:36:10.787155: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e25d00 of size 2048 next 168 2025-05-15 06:36:10.787167: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e26500 of size 2048 next 169 2025-05-15 06:36:10.787179: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e26d00 of size 2048 next 170 2025-05-15 06:36:10.787191: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e27500 of size 1024 next 172 2025-05-15 06:36:10.787203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e27900 of size 1024 next 173 2025-05-15 06:36:10.787215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e27d00 of size 1024 next 174 2025-05-15 06:36:10.787227: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e28100 of size 1024 next 175 2025-05-15 06:36:10.787239: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e28500 of size 1024 next 176 2025-05-15 06:36:10.787251: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e28900 of size 256 next 178 2025-05-15 06:36:10.787277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e28a00 of size 256 next 179 2025-05-15 06:36:10.787289: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e28b00 of size 1024 next 177 2025-05-15 06:36:10.787301: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e28f00 of size 1024 next 182 2025-05-15 06:36:10.787313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e29300 of size 1024 next 183 2025-05-15 06:36:10.787325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e29700 of size 1024 next 184 2025-05-15 06:36:10.787337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e29b00 of size 1024 next 185 2025-05-15 06:36:10.787349: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e29f00 of size 256 next 187 2025-05-15 06:36:10.787361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e2a000 of size 256 next 188 2025-05-15 06:36:10.787373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e2a100 of size 4096 next 186 2025-05-15 06:36:10.787384: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e2b100 of size 4096 next 42 2025-05-15 06:36:10.787396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e2c100 of size 65536 next 36 2025-05-15 06:36:10.787408: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e3c100 of size 65536 next 35 2025-05-15 06:36:10.787420: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e4c100 of size 4096 next 235 2025-05-15 06:36:10.787432: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e4d100 of size 4096 next 236 2025-05-15 06:36:10.787444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e4e100 of size 4096 next 237 2025-05-15 06:36:10.787456: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e4f100 of size 4096 next 238 2025-05-15 06:36:10.787468: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e50100 of size 1024 next 239 2025-05-15 06:36:10.787480: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e50500 of size 1024 next 240 2025-05-15 06:36:10.787491: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e50900 of size 1024 next 241 2025-05-15 06:36:10.787503: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e50d00 of size 1024 next 242 2025-05-15 06:36:10.787515: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e51100 of size 1024 next 243 2025-05-15 06:36:10.787527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e51500 of size 1024 next 246 2025-05-15 06:36:10.787539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e51900 of size 1024 next 247 2025-05-15 06:36:10.787550: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e51d00 of size 1024 next 248 2025-05-15 06:36:10.787562: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e52100 of size 1024 next 249 2025-05-15 06:36:10.787574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e52500 of size 1024 next 250 2025-05-15 06:36:10.787586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e52900 of size 4096 next 252 2025-05-15 06:36:10.787598: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e53900 of size 4096 next 253 2025-05-15 06:36:10.787610: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e54900 of size 4096 next 254 2025-05-15 06:36:10.787622: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e55900 of size 4096 next 255 2025-05-15 06:36:10.787633: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e56900 of size 4096 next 256 2025-05-15 06:36:10.787645: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e57900 of size 1024 next 257 2025-05-15 06:36:10.787662: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e57d00 of size 1024 next 258 2025-05-15 06:36:10.787670: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e58100 of size 1024 next 259 2025-05-15 06:36:10.787679: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e58500 of size 1024 next 260 2025-05-15 06:36:10.787687: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e58900 of size 1024 next 261 2025-05-15 06:36:10.787696: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e58d00 of size 1024 next 264 2025-05-15 06:36:10.787704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e59100 of size 1024 next 265 2025-05-15 06:36:10.787713: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e59500 of size 1024 next 266 2025-05-15 06:36:10.787721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e59900 of size 1024 next 267 2025-05-15 06:36:10.787729: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e59d00 of size 1024 next 268 2025-05-15 06:36:10.787738: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5a100 of size 4096 next 270 2025-05-15 06:36:10.787746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5b100 of size 4096 next 271 2025-05-15 06:36:10.787755: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5c100 of size 4096 next 272 2025-05-15 06:36:10.787763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5d100 of size 4096 next 273 2025-05-15 06:36:10.787772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5e100 of size 4096 next 274 2025-05-15 06:36:10.787780: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5f100 of size 1024 next 275 2025-05-15 06:36:10.787789: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5f500 of size 1024 next 276 2025-05-15 06:36:10.787797: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5f900 of size 1024 next 277 2025-05-15 06:36:10.787805: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e5fd00 of size 1024 next 278 2025-05-15 06:36:10.787814: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e60100 of size 1024 next 279 2025-05-15 06:36:10.787822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e60500 of size 1024 next 282 2025-05-15 06:36:10.787831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e60900 of size 1024 next 283 2025-05-15 06:36:10.787839: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e60d00 of size 1024 next 284 2025-05-15 06:36:10.787848: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e61100 of size 1024 next 285 2025-05-15 06:36:10.787856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e61500 of size 1024 next 286 2025-05-15 06:36:10.787864: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e61900 of size 4096 next 288 2025-05-15 06:36:10.787873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e62900 of size 6144 next 27 2025-05-15 06:36:10.787882: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e64100 of size 147456 next 26 2025-05-15 06:36:10.787890: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e88100 of size 65536 next 47 2025-05-15 06:36:10.787899: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990e98100 of size 65536 next 59 2025-05-15 06:36:10.787907: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990ea8100 of size 65536 next 65 2025-05-15 06:36:10.787916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7990eb8100 of size 294656 next 18446744073709551615 2025-05-15 06:36:10.787925: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-05-15 06:36:10.787945: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 65 Chunks of size 256 totalling 16.2KiB 2025-05-15 06:36:10.787956: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-05-15 06:36:10.787966: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 250 Chunks of size 1024 totalling 250.0KiB 2025-05-15 06:36:10.787975: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-05-15 06:36:10.787985: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 2048 totalling 80.0KiB 2025-05-15 06:36:10.787994: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 117 Chunks of size 4096 totalling 468.0KiB 2025-05-15 06:36:10.788004: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 6144 totalling 12.0KiB 2025-05-15 06:36:10.788013: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2025-05-15 06:36:10.788023: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 10 Chunks of size 8192 totalling 80.0KiB 2025-05-15 06:36:10.788033: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-05-15 06:36:10.788042: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-05-15 06:36:10.788052: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-05-15 06:36:10.788061: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-05-15 06:36:10.788071: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-05-15 06:36:10.788080: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2025-05-15 06:36:10.788090: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-05-15 06:36:10.788099: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-05-15 06:36:10.788108: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2025-05-15 06:36:10.788118: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-05-15 06:36:10.788127: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-05-15 06:36:10.788137: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1048576 totalling 25.00MiB 2025-05-15 06:36:10.788146: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 18 Chunks of size 1310720 totalling 22.50MiB 2025-05-15 06:36:10.788156: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB 2025-05-15 06:36:10.788165: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB 2025-05-15 06:36:10.788174: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 2097152 totalling 4.00MiB 2025-05-15 06:36:10.788184: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 23 Chunks of size 2359296 totalling 51.75MiB 2025-05-15 06:36:10.788193: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 4194304 totalling 8.00MiB 2025-05-15 06:36:10.788203: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 8388608 totalling 8.00MiB 2025-05-15 06:36:10.788212: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 9437184 totalling 18.00MiB 2025-05-15 06:36:10.788222: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 13500416 totalling 12.88MiB 2025-05-15 06:36:10.788231: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 160.48MiB 2025-05-15 06:36:10.788240: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 201195520 memory_limit_: 201195520 available bytes: 0 curr_region_allocation_bytes_: 268435456 2025-05-15 06:36:10.788261: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 201195520 InUse: 168275456 MaxInUse: 168275712 NumAllocs: 2113 MaxAllocSize: 13500416 2025-05-15 06:36:10.788287: W tensorflow/core/common_runtime/bfc_allocator.cc:439] __*****__******___**************x*********************************_************************_******** 2025-05-15 06:36:10.789160: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 ERROR : 'int' object is not subscriptable reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: "'int' object is not subscriptable" /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_6b55ca732dc5789dd54794d418371b3abe63c01d SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_6b55ca732dc5789dd54794d418371b3abe63c01d','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/May/15052025/python_test3//data_2/data_log/job/2025/May/15052025/python_test3/log-python3----short_python3--v--marlene-06:35:02.txt','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.2149333953857422 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Thu May 15 06:36:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747283771_1979549_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1747283771_1979549_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 ERROR in datou_step_exec, will save and exit ! CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2430, in datou_step_exec return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 367, in datou_step_sam sam.to(device=device) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to return self._apply(convert) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 602, in _apply param_applied = fn(param) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', "[>, , , , , 'CUDA error: out of memory\\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.015873193740844727 save_final ERROR in last step sam, CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. time spend for datou_step_exec : 5.770626783370972 time spend to save output : 0.03239321708679199 total time spend for step 0 : 5.803020000457764 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : None ERROR nb objects espect : 98 nb_objects detect : 0 ERROR sam FAILED ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.13892650604248047 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Thu May 15 06:36:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747283777_1979549_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1747283777_1979549_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] local folder : /data/models_weight/detection_plaque_valcor_010622 /data/models_weight/detection_plaque_valcor_010622/caffemodel size_local : 349723073 size in s3 : 349723073 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:56 caffemodel already exist and didn't need to update /data/models_weight/detection_plaque_valcor_010622/test.prototxt size_local : 7163 size in s3 : 7163 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:55 test.prototxt already exist and didn't need to update prototxt : /data/models_weight/detection_plaque_valcor_010622/test.prototxt caffemodel : /data/models_weight/detection_plaque_valcor_010622/caffemodel Loaded network /data/models_weight/detection_plaque_valcor_010622/caffemodel About to compute detect_faster_rcnn : len(args) : 1 Inside frcnn step exec : nb paths : 1 image_path : temp/1747283777_1979549_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg image_size (600, 800, 3) [[[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] ... [[ 14 16 16] [ 13 15 15] [ 11 13 13] ... [198 206 205] [198 206 205] [198 206 205]] [[ 16 18 18] [ 14 16 16] [ 11 13 13] ... [206 214 213] [206 214 213] [206 214 213]] [[ 13 15 15] [ 12 14 14] [ 9 11 11] ... [210 218 217] [210 218 217] [210 218 217]]] Detection took 0.081s for 300 object proposals c : plaque list_crops.shape (72, 5) proba : 0.06384314 (374.12698, 293.9179, 430.81064, 317.80756) proba : 0.05222075 (382.17822, 297.18857, 552.36053, 344.658) proba : 0.012273547 (345.3573, 272.4279, 468.8562, 320.7231) We are managing local photo_id len de result frcnn : 1 After datou_step_exec type output : time spend for datou_step_exec : 3.315631628036499 time spend to save output : 0.00013136863708496094 total time spend for step 1 : 3.315762996673584 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True Inside saveFrcnn : final : True verbose : True threshold to save the result : 0.1 output flattener : [(0, 493029425, 4370, 374, 430, 293, 317, 0.06384314, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.05222075, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012273547, None)] Warning : no hashtag_ids to insert in the database final : True begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', '0', 0, '0', 493061979, '0', None)] time used for this insertion : 0.013655424118041992 [917754606] map_info['map_portfolio_photo'] : {} final : True mtd_id 4184 list_pids : [917754606] Looping around the photos to save general results len do output : 1 /0 before output type Managing all output in save final without adding information in the mtr_datou_result ('4184', None, None, None, None, None, None, None, None) ('4184', None, '917754606', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', None, None, None, None, None, None)] time used for this insertion : 0.012830734252929688 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {0: [[(0, 493029425, 4370, 374, 430, 293, 317, 0.06384314, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.05222075, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012273547, None)], 'temp/1747283777_1979549_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg']} ############################### TEST thcl ################################ TEST THCL Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=2 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=2 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 2 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=2 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 1 thcl is not linked in the step_by_step architecture ! WARNING : step 2 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (916235064) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 916235064 download finish for photo 916235064 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.10057711601257324 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:thcl Thu May 15 06:36:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747283780_1979549_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1747283780_1979549_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'355': 1} we are using the classfication for only one thcl 355 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (66, 66, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.009034872055053711 time to convert the images to numpy array : 0.003522157669067383 total time to convert the images to numpy array : 0.012926816940307617 list photo_ids error: [] list photo_ids correct : [916235064] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 355 To do loadFromThcl(), then load ParamDescType : thcl355 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (355) thcls : [{'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3390 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) To loadFromThcl() : net_3390 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3154 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) param : , param.caffemodel : car_360_1027 None mean_file_type : mean_file_path : prototxt_file_path : model : car_360_1027 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : car_360_1027 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/car_360_1027 /data/models_weight/car_360_1027/caffemodel size_local : 542944640 size in s3 : 542944640 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 caffemodel already exist and didn't need to update /data/models_weight/car_360_1027/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy_fc.prototxt size_local : 1132 size in s3 : 1132 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy.prototxt size_local : 5654 size in s3 : 5654 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/mean.npy size_local : 1572944 size in s3 : 1572944 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:55 mean.npy already exist and didn't need to update /data/models_weight/car_360_1027/synset_words.txt size_local : 13687 size in s3 : 13687 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/car_360_1027/deploy.prototxt caffemodel_filename : /data/models_weight/car_360_1027/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3154 max_wait_temp : 1 max_wait : 0 dict_keys(['prob', 'pool5']) time used to do the prepocess of the images : 0.01385641098022461 time used to do the prediction : 0.08869504928588867 save descriptor for thcl : 355 (1, 512, 7, 7) Got the blobs of the net to insert : [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] code_as_byte_string:b'0000000000'| time to traite the descriptors : 0.07923507690429688 Testing : ['916235064'] In select_photos_meta_from_ids: SELECT photo_id, url, FROM_UNIXTIME(uploaded_at), latitude, longitude, text FROM MTRBack.photos WHERE photo_id IN (916235064) result : {916235064: {'photo_id': 916235064, 'url': 'https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/2017/10/14/6293d1bb790dc6902450e7c572b7d10b.jpg', 'latitude': 0.0, 'longitude': 0.0, 'text': None}} list_photo_exists : [916235064] storage_type for insertDescriptorsMulti : 1 To insert : 916235064 time to insert the descriptors : 0.6117620468139648 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : False verbose : True time used to find the portfolios of the photos select button_legend_list from MTRDatou.classification_theme where id = 355 SAVE THCL, output : {'916235064': [[('916235064', 'c_elysee_1027_gao__port_506302', 0.0018815873, 332, '355'), ('916235064', 'mokka_1027_gao__port_506374', 0.0011635823, 332, '355'), ('916235064', 'captur_1027_gao__port_506399', 0.0008158261, 332, '355'), ('916235064', 'sorento_1027_gao__port_506192', 0.0011773275, 332, '355'), ('916235064', 'navara_1027_gao__port_506205', 0.0025850255, 332, '355'), ('916235064', 'xc90_1027_gao__port_506350', 0.0041701803, 332, '355'), ('916235064', 'saxo_1027_gao__port_506052', 0.0034809185, 332, '355'), ('916235064', 'trafic_1027_gao__port_506295', 0.0073665935, 332, '355'), ('916235064', 'punto_evo_1027_gao__port_506066', 0.0021887447, 332, '355'), ('916235064', '5_1027_gao__port_506117', 0.0005797829, 332, '355'), ('916235064', '250_1027_gao__port_506065', 0.0045911814, 332, '355'), ('916235064', 'd_max_1027_gao__port_506125', 0.0031585875, 332, '355'), ('916235064', 'panamera_1027_gao__port_506387', 0.0022508057, 332, '355'), ('916235064', 'alhambra_1027_gao__port_506381', 0.005320156, 332, '355'), ('916235064', 'x6_1027_gao__port_506349', 0.001099895, 332, '355'), ('916235064', 'vitara_1027_gao__port_506328', 0.0054023056, 332, '355'), ('916235064', 'fiesta_1027_gao__port_506377', 0.003919088, 332, '355'), ('916235064', 'qashqai_1027_gao__port_506286', 0.0014787901, 332, '355'), ('916235064', '147_1027_gao__port_506124', 0.0019777678, 332, '355'), ('916235064', 'c5_1027_gao__port_506172', 0.0012441734, 332, '355'), ('916235064', 'q5_1027_gao__port_506206', 0.0015050189, 332, '355'), ('916235064', 'giulia_1027_gao__port_506178', 0.002169433, 332, '355'), ('916235064', 'karl_1027_gao__port_506371', 0.0027081112, 332, '355'), ('916235064', 'mehari_1027_gao__port_506076', 0.0047036065, 332, '355'), ('916235064', '911_1027_gao__port_506114', 0.001941943, 332, '355'), ('916235064', '508_1027_gao__port_506329', 0.00095857313, 332, '355'), ('916235064', 'idea_1027_gao__port_506122', 0.00076998066, 332, '355'), ('916235064', 'megane_1027_gao__port_506220', 0.0019468918, 332, '355'), ('916235064', 'ghibli_1027_gao__port_506174', 0.0013725258, 332, '355'), ('916235064', 'touareg_1027_gao__port_506224', 0.0016202376, 332, '355'), ('916235064', 'i10_1027_gao__port_506232', 0.0013925241, 332, '355'), ('916235064', 'jumper_1027_gao__port_506234', 0.010043544, 332, '355'), ('916235064', 'classe_clk_1027_gao__port_506173', 0.0010793952, 332, '355'), ('916235064', 'kuga_1027_gao__port_506181', 0.0008447126, 332, '355'), ('916235064', 'ct_1027_gao__port_506323', 0.001252063, 332, '355'), ('916235064', 'leon_1027_gao__port_506326', 0.0025845533, 332, '355'), ('916235064', 'ds5_1027_gao__port_506376', 0.001243074, 332, '355'), ('916235064', 'cordoba_1027_gao__port_506048', 0.0028650449, 332, '355'), ('916235064', 'classe_cla_1027_gao__port_506400', 0.0012950172, 332, '355'), ('916235064', 'jumpy_1027_gao__port_506179', 0.010337893, 332, '355'), ('916235064', 'avensis_1027_gao__port_506311', 0.0018767836, 332, '355'), ('916235064', 'juke_1027_gao__port_506325', 0.0011344035, 332, '355'), ('916235064', '4008_1027_gao__port_506402', 0.0015758829, 332, '355'), ('916235064', '190_series_1027_gao__port_506051', 0.0039804503, 332, '355'), ('916235064', 'serie_3_1027_gao__port_506294', 0.0028741332, 332, '355'), ('916235064', 'q7_1027_gao__port_506318', 0.0023355589, 332, '355'), ('916235064', 'glc_1027_gao__port_506303', 0.0012107086, 332, '355'), ('916235064', 'grand_vitara_1027_gao__port_506175', 0.001144763, 332, '355'), ('916235064', 's40_1027_gao__port_506099', 0.002233956, 332, '355'), ('916235064', 'toledo_1027_gao__port_506061', 0.0017464566, 332, '355'), ('916235064', '5008_1027_gao__port_506337', 0.004699305, 332, '355'), ('916235064', 'continental_1027_gao__port_506250', 0.0021914812, 332, '355'), ('916235064', 'coupe_1027_gao__port_506082', 0.0022632761, 332, '355'), ('916235064', 'iq_1027_gao__port_506166', 0.001817427, 332, '355'), ('916235064', '407_1027_gao__port_506133', 0.0009056769, 332, '355'), ('916235064', 'touran_1027_gao__port_506308', 0.0020402165, 332, '355'), ('916235064', '300c_1027_gao__port_506078', 0.0025336416, 332, '355'), ('916235064', 'classe_gl_1027_gao__port_506340', 0.00448914, 332, '355'), ('916235064', 'vivaro_1027_gao__port_506310', 0.003425301, 332, '355'), ('916235064', 'sl_1027_gao__port_506100', 0.0031355126, 332, '355'), ('916235064', 'elise_1027_gao__port_506121', 0.001025605, 332, '355'), ('916235064', '1007_1027_gao__port_506070', 0.001535401, 332, '355'), ('916235064', 'i40_1027_gao__port_506218', 0.0005915174, 332, '355'), ('916235064', 'bipper_tepee_1027_gao__port_506227', 0.0040291147, 332, '355'), ('916235064', 'focus_1027_gao__port_506272', 0.0011586425, 332, '355'), ('916235064', 'primera_1027_gao__port_506147', 0.0012158762, 332, '355'), ('916235064', 'r4_1027_gao__port_506160', 0.014966348, 332, '355'), ('916235064', 'a8_1027_gao__port_506265', 0.0011321139, 332, '355'), ('916235064', 'boxer_1027_gao__port_506202', 0.010545176, 332, '355'), ('916235064', 's5_1027_gao__port_506222', 0.0011985628, 332, '355'), ('916235064', 'r21_1027_gao__port_506093', 0.004185765, 332, '355'), ('916235064', 'c3_1027_gao__port_506257', 0.0023635477, 332, '355'), ('916235064', 'santa_fe_1027_gao__port_506208', 0.001632395, 332, '355'), ('916235064', 'm4_1027_gao__port_506344', 0.0015568209, 332, '355'), ('916235064', 'safrane_1027_gao__port_506077', 0.0013959236, 332, '355'), ('916235064', 'classe_gle_1027_gao__port_506395', 0.002197987, 332, '355'), ('916235064', '0_1027_gao__port_506094', 0.008827525, 332, '355'), ('916235064', 'ix35_1027_gao__port_506219', 0.0014614736, 332, '355'), ('916235064', 'carens_1027_gao__port_506298', 0.00088249805, 332, '355'), ('916235064', 'classe_a_1027_gao__port_506339', 0.002471451, 332, '355'), ('916235064', 'ix20_1027_gao__port_506343', 0.0010092582, 332, '355'), ('916235064', 'note_1027_gao__port_506365', 0.001596198, 332, '355'), ('916235064', 'a5_1027_gao__port_506200', 0.0015330885, 332, '355'), ('916235064', 'sx4_1027_gao__port_506348', 0.0014917415, 332, '355'), ('916235064', 'sandero_1027_gao__port_506198', 0.0014585756, 332, '355'), ('916235064', '3008_1027_gao__port_506385', 0.0056460095, 332, '355'), ('916235064', 'q50_1027_gao__port_506239', 0.0011165747, 332, '355'), ('916235064', 'latitude_1027_gao__port_506236', 0.0008019638, 332, '355'), ('916235064', 'v40_1027_gao__port_506391', 0.001714722, 332, '355'), ('916235064', 'xsara_1027_gao__port_506087', 0.0009823057, 332, '355'), ('916235064', 'grand_c_max_1027_gao__port_506342', 0.0017957998, 332, '355'), ('916235064', 'swift_1027_gao__port_506149', 0.0015020518, 332, '355'), ('916235064', 'serie_1_1027_gao__port_506184', 0.0015139554, 332, '355'), ('916235064', 'xc70_1027_gao__port_506393', 0.0036194003, 332, '355'), ('916235064', 'master_1027_gao__port_506203', 0.007957451, 332, '355'), ('916235064', 'clio_1027_gao__port_506280', 0.0029575205, 332, '355'), ('916235064', 'duster_1027_gao__port_506216', 0.00074439053, 332, '355'), ('916235064', 'traveller_1027_gao__port_506403', 0.0042939964, 332, '355'), ('916235064', 'tipo_1027_gao__port_506355', 0.0010929515, 332, '355'), ('916235064', 'rav_4_1027_gao__port_506332', 0.0013604126, 332, '355'), ('916235064', 'coccinelle_1027_gao__port_506259', 0.003494821, 332, '355'), ('916235064', 'spacetourer_1027_gao__port_506401', 0.0030972431, 332, '355'), ('916235064', 'xe_1027_gao__port_506357', 0.0014471379, 332, '355'), ('916235064', 'ds3_1027_gao__port_506324', 0.0013093834, 332, '355'), ('916235064', 'mx_5_1027_gao__port_506098', 0.0025887527, 332, '355'), ('916235064', 'land_cruiser_1027_gao__port_506315', 0.009531158, 332, '355'), ('916235064', 'classe_b_1027_gao__port_506335', 0.0017216237, 332, '355'), ('916235064', '806_1027_gao__port_506088', 0.0025618088, 332, '355'), ('916235064', 'rx_8_1027_gao__port_506046', 0.0036219028, 332, '355'), ('916235064', 'spark_1027_gao__port_506185', 0.0010077776, 332, '355'), ('916235064', '6_1027_gao__port_506171', 0.0011183119, 332, '355'), ('916235064', 'bravo_1027_gao__port_506080', 0.0014650166, 332, '355'), ('916235064', 'nx_1027_gao__port_506345', 0.0013683371, 332, '355'), ('916235064', 'sharan_1027_gao__port_506347', 0.00509229, 332, '355'), ('916235064', 'x_type_1027_gao__port_506067', 0.00078029645, 332, '355'), ('916235064', 'jimny_1027_gao__port_506233', 0.004605893, 332, '355'), ('916235064', 'wrangler_1027_gao__port_506225', 0.0017997316, 332, '355'), ('916235064', 'c_crosser_1027_gao__port_506312', 0.0015926675, 332, '355'), ('916235064', 'v70_1027_gao__port_506278', 0.0019676946, 332, '355'), ('916235064', 'classe_e_1027_gao__port_506300', 0.0017369459, 332, '355'), ('916235064', 'classe_v_1027_gao__port_506258', 0.0127302855, 332, '355'), ('916235064', 'm3_1027_gao__port_506182', 0.0023371254, 332, '355'), ('916235064', 'abarth_500_1027_gao__port_506226', 0.0040435344, 332, '355'), ('916235064', 'serie_6_1027_gao__port_506262', 0.0011315026, 332, '355'), ('916235064', 'modus_1027_gao__port_506146', 0.0018292351, 332, '355'), ('916235064', '3_1027_gao__port_506113', 0.0015082406, 332, '355'), ('916235064', '405_1027_gao__port_506108', 0.008055281, 332, '355'), ('916235064', 'allroad_1027_gao__port_506297', 0.0010598102, 332, '355'), ('916235064', 'auris_1027_gao__port_506322', 0.001152502, 332, '355'), ('916235064', 'galaxy_1027_gao__port_506143', 0.0025147826, 332, '355'), ('916235064', 'giulietta_1027_gao__port_506363', 0.0008651137, 332, '355'), ('916235064', '106_1027_gao__port_506073', 0.008270407, 332, '355'), ('916235064', 'classe_m_1027_gao__port_506154', 0.0030020282, 332, '355'), ('916235064', 'espace_1027_gao__port_506313', 0.0010645925, 332, '355'), ('916235064', 'panda_1027_gao__port_506189', 0.009029626, 332, '355'), ('916235064', 'rcz_1027_gao__port_506197', 0.0011294273, 332, '355'), ('916235064', '4007_1027_gao__port_506162', 0.00067927473, 332, '355'), ('916235064', 'classe_cl_1027_gao__port_506249', 0.0010861, 332, '355'), ('916235064', 'leaf_1027_gao__port_506139', 0.0018038912, 332, '355'), ('916235064', 'octavia_1027_gao__port_506237', 0.0018601883, 332, '355'), ('916235064', 'ds4_1027_gao__port_506336', 0.002415927, 332, '355'), ('916235064', 'freelander_1027_gao__port_506084', 0.002347437, 332, '355'), ('916235064', 'evasion_1027_gao__port_506109', 0.0031140149, 332, '355'), ('916235064', 'punto_1027_gao__port_506106', 0.0019495742, 332, '355'), ('916235064', '2cv_1027_gao__port_506045', 0.007974094, 332, '355'), ('916235064', 'x4_1027_gao__port_506392', 0.001794904, 332, '355'), ('916235064', 'antara_1027_gao__port_506247', 0.0012469144, 332, '355'), ('916235064', 'murano_1027_gao__port_506316', 0.00060889253, 332, '355'), ('916235064', 'alto_1027_gao__port_506201', 0.009231365, 332, '355'), ('916235064', 'meriva_1027_gao__port_506353', 0.0013764178, 332, '355'), ('916235064', 'orlando_1027_gao__port_506305', 0.00184606, 332, '355'), ('916235064', 'new_beetle_1027_gao__port_506050', 0.0011637948, 332, '355'), ('916235064', '306_1027_gao__port_506145', 0.0035072144, 332, '355'), ('916235064', 'tiguan_1027_gao__port_506362', 0.0026825822, 332, '355'), ('916235064', 's_type_1027_gao__port_506101', 0.0011382444, 332, '355'), ('916235064', 'c1_1027_gao__port_506128', 0.0027514927, 332, '355'), ('916235064', 'vectra_1027_gao__port_506044', 0.001198979, 332, '355'), ('916235064', 'outlander_1027_gao__port_506317', 0.0017121345, 332, '355'), ('916235064', '307_1027_gao__port_506074', 0.0020011603, 332, '355'), ('916235064', 'a6_s6_1027_gao__port_506134', 0.0016570878, 332, '355'), ('916235064', 'nemo_combi_1027_gao__port_506196', 0.0022661795, 332, '355'), ('916235064', 'berlingo_1027_gao__port_506194', 0.0046624746, 332, '355'), ('916235064', 'partner_1027_gao__port_506285', 0.0039411546, 332, '355'), ('916235064', 'cayenne_1027_gao__port_506177', 0.0037980105, 332, '355'), ('916235064', 'quattroporte_1027_gao__port_506240', 0.0024440566, 332, '355'), ('916235064', 'c_max_1027_gao__port_506282', 0.0013124436, 332, '355'), ('916235064', 'fabia_1027_gao__port_506396', 0.0052991514, 332, '355'), ('916235064', 'cx_3_1027_gao__port_506281', 0.0014462498, 332, '355'), ('916235064', 'x_trail_1027_gao__port_506264', 0.0018315319, 332, '355'), ('916235064', 'scirocco_1027_gao__port_506276', 0.0047907243, 332, '355'), ('916235064', 'matiz_1027_gao__port_506144', 0.0017560543, 332, '355'), ('916235064', 'tigra_1027_gao__port_506069', 0.0008542583, 332, '355'), ('916235064', 'escort_1027_gao__port_506091', 0.004840251, 332, '355'), ('916235064', 'c2_1027_gao__port_506081', 0.0014904155, 332, '355'), ('916235064', 'mini_1027_gao__port_506168', 0.0011921772, 332, '355'), ('916235064', 'i30_1027_gao__port_506291', 0.00063257746, 332, '355'), ('916235064', 'picanto_1027_gao__port_506238', 0.0029747407, 332, '355'), ('916235064', 'mito_1027_gao__port_506072', 0.0015077492, 332, '355'), ('916235064', 'impreza_1027_gao__port_506085', 0.0020191905, 332, '355'), ('916235064', 'kangoo_1027_gao__port_506235', 0.006583785, 332, '355'), ('916235064', 'a4_1027_gao__port_506193', 0.0019875139, 332, '355'), ('916235064', 'cayman_1027_gao__port_506268', 0.0018140603, 332, '355'), ('916235064', 'sportage_1027_gao__port_506148', 0.0014275927, 332, '355'), ('916235064', 'up_1027_gao__port_506356', 0.0068632527, 332, '355'), ('916235064', 'optima_1027_gao__port_506386', 0.0008918806, 332, '355'), ('916235064', 'defender_1027_gao__port_506229', 0.006721449, 332, '355'), ('916235064', 'serie_2_1027_gao__port_506256', 0.0022274463, 332, '355'), ('916235064', 'edge_1027_gao__port_506187', 0.000874862, 332, '355'), ('916235064', 'r19_1027_gao__port_506110', 0.004942707, 332, '355'), ('916235064', 'jetta_1027_gao__port_506304', 0.0036194762, 332, '355'), ('916235064', 'eos_1027_gao__port_506115', 0.0038924438, 332, '355'), ('916235064', 'accord_1027_gao__port_506214', 0.002012817, 332, '355'), ('916235064', 'yaris_1027_gao__port_506334', 0.0032334228, 332, '355'), ('916235064', 'classe_cls_1027_gao__port_506289', 0.0007852072, 332, '355'), ('916235064', 'polo_1027_gao__port_506361', 0.0043109255, 332, '355'), ('916235064', 'serie_4_1027_gao__port_506366', 0.0011475991, 332, '355'), ('916235064', 'mini_cabriolet_1027_gao__port_506204', 0.00083778286, 332, '355'), ('916235064', 'prius_1027_gao__port_506190', 0.0011489403, 332, '355'), ('916235064', 'lodgy_1027_gao__port_506188', 0.002017252, 332, '355'), ('916235064', 'serie_7_1027_gao__port_506307', 0.0012477316, 332, '355'), ('916235064', 'c15_1027_gao__port_506055', 0.017712925, 332, '355'), ('916235064', 'kadjar_1027_gao__port_506389', 0.0012505528, 332, '355'), ('916235064', 'insignia_1027_gao__port_506364', 0.0016434424, 332, '355'), ('916235064', '308_1027_gao__port_506279', 0.0021236858, 332, '355'), ('916235064', 'roomster_1027_gao__port_506241', 0.0018012671, 332, '355'), ('916235064', '80_1027_gao__port_506057', 0.0046137194, 332, '355'), ('916235064', '309_1027_gao__port_506063', 0.013525366, 332, '355'), ('916235064', 'tucson_1027_gao__port_506320', 0.0021236797, 332, '355'), ('916235064', 'x3_1027_gao__port_506212', 0.00089766574, 332, '355'), ('916235064', 'xf_1027_gao__port_506263', 0.0011166116, 332, '355'), ('916235064', '2008_1027_gao__port_506394', 0.0026409077, 332, '355'), ('916235064', 'passat_1027_gao__port_506306', 0.0014976945, 332, '355'), ('916235064', 'compass_1027_gao__port_506260', 0.0032558222, 332, '355'), ('916235064', 'twingo_1027_gao__port_506309', 0.006499071, 332, '355'), ('916235064', 'micra_1027_gao__port_506221', 0.0035859174, 332, '355'), ('916235064', 'golf_1027_gao__port_506155', 0.0031932972, 332, '355'), ('916235064', 'soul_1027_gao__port_506176', 0.0012889629, 332, '355'), ('916235064', 'rapid_1027_gao__port_506398', 0.0025915501, 332, '355'), ('916235064', 'forester_1027_gao__port_506360', 0.0022762427, 332, '355'), ('916235064', 'slk_1027_gao__port_506210', 0.0015845589, 332, '355'), ('916235064', 'forfour_1027_gao__port_506341', 0.002179723, 332, '355'), ('916235064', 'serie_5_1027_gao__port_506209', 0.0013695332, 332, '355'), ('916235064', 'xj_1027_gao__port_506170', 0.0026009479, 332, '355'), ('916235064', 'pajero_1027_gao__port_506097', 0.0052111535, 332, '355'), ('916235064', 'agila_1027_gao__port_506119', 0.004849574, 332, '355'), ('916235064', 'a6_1027_gao__port_506163', 0.0018911443, 332, '355'), ('916235064', 'fox_1027_gao__port_506092', 0.00084655, 332, '355'), ('916235064', 'boxster_1027_gao__port_506267', 0.0015942679, 332, '355'), ('916235064', 'altea_1027_gao__port_506246', 0.0021391062, 332, '355'), ('916235064', 'samurai_1027_gao__port_506047', 0.0062514143, 332, '355'), ('916235064', 'trax_1027_gao__port_506296', 0.0019440071, 332, '355'), ('916235064', 'getz_1027_gao__port_506058', 0.0016385034, 332, '355'), ('916235064', 'cherokee_1027_gao__port_506269', 0.002981103, 332, '355'), ('916235064', 'koleos_1027_gao__port_506378', 0.0015591882, 332, '355'), ('916235064', 'z_series_1027_gao__port_506123', 0.0016563401, 332, '355'), ('916235064', 'ecosport_1027_gao__port_506271', 0.0013230463, 332, '355'), ('916235064', 'space_star_1027_gao__port_506277', 0.0021144378, 332, '355'), ('916235064', 'rs3_sportback_1027_gao__port_506207', 0.0019117557, 332, '355'), ('916235064', 'civic_1027_gao__port_506141', 0.0026900237, 332, '355'), ('916235064', 'talisman_1027_gao__port_506390', 0.0007613836, 332, '355'), ('916235064', 'f_pace_1027_gao__port_506314', 0.0016166099, 332, '355'), ('916235064', 'classe_c_1027_gao__port_506299', 0.0017941976, 332, '355'), ('916235064', 'tt_1027_gao__port_506075', 0.0013935039, 332, '355'), ('916235064', 'pathfinder_1027_gao__port_506183', 0.0016513828, 332, '355'), ('916235064', '156_1027_gao__port_506157', 0.0015444162, 332, '355'), ('916235064', 'cx_5_1027_gao__port_506228', 0.0014413547, 332, '355'), ('916235064', 'scenic_1027_gao__port_506255', 0.0016083972, 332, '355'), ('916235064', 'yeti_1027_gao__port_506358', 0.0020915105, 332, '355'), ('916235064', 'mustang_1027_gao__port_506053', 0.01005022, 332, '355'), ('916235064', 'stilo_1027_gao__port_506060', 0.0010831953, 332, '355'), ('916235064', 'ateca_1027_gao__port_506382', 0.0017011532, 332, '355'), ('916235064', 'fiorino_1027_gao__port_506217', 0.009198338, 332, '355'), ('916235064', 'classe_glk_1027_gao__port_506290', 0.0017017283, 332, '355'), ('916235064', 'fortwo_1027_gao__port_506230', 0.0016010152, 332, '355'), ('916235064', 'cruze_1027_gao__port_506186', 0.0010053078, 332, '355'), ('916235064', '107_1027_gao__port_506213', 0.0016276134, 332, '355'), ('916235064', 'aygo_1027_gao__port_506248', 0.0032434769, 332, '355'), ('916235064', 'rx_1027_gao__port_506354', 0.0010632961, 332, '355'), ('916235064', '500_1027_gao__port_506245', 0.0016354569, 332, '355'), ('916235064', 'bora_1027_gao__port_506104', 0.0038165813, 332, '355'), ('916235064', 'transit_1027_gao__port_506111', 0.004861858, 332, '355'), ('916235064', 'pt_cruiser_1027_gao__port_506054', 0.0019163836, 332, '355'), ('916235064', 'patrol_1027_gao__port_506068', 0.0042399126, 332, '355'), ('916235064', 'r8_1027_gao__port_506156', 0.0012722596, 332, '355'), ('916235064', 'xm_1027_gao__port_506102', 0.0022680133, 332, '355'), ('916235064', 's60_1027_gao__port_506191', 0.0031992958, 332, '355'), ('916235064', 'aveo_1027_gao__port_506158', 0.003839947, 332, '355'), ('916235064', 'captiva_1027_gao__port_506159', 0.0017188658, 332, '355'), ('916235064', 'ax_1027_gao__port_506153', 0.006896724, 332, '355'), ('916235064', 'rexton_1027_gao__port_506107', 0.0013021142, 332, '355'), ('916235064', 'camaro_1027_gao__port_506056', 0.0024903647, 332, '355'), ('916235064', 'ypsilon_1027_gao__port_506131', 0.0019541448, 332, '355'), ('916235064', 'delta_1027_gao__port_506165', 0.0014002038, 332, '355'), ('916235064', 'c4_1027_gao__port_506370', 0.001300941, 332, '355'), ('916235064', 'zx_1027_gao__port_506161', 0.0045941095, 332, '355'), ('916235064', 'verso_1027_gao__port_506242', 0.00077212264, 332, '355'), ('916235064', 'superb_1027_gao__port_506327', 0.0019941751, 332, '355'), ('916235064', 'r5_1027_gao__port_506253', 0.0095462985, 332, '355'), ('916235064', 'caddy_1027_gao__port_506330', 0.013825928, 332, '355'), ('916235064', 'x5_1027_gao__port_506243', 0.0011203894, 332, '355'), ('916235064', 'f_type_1027_gao__port_506231', 0.00082998106, 332, '355'), ('916235064', 'fusion_1027_gao__port_506096', 0.001266988, 332, '355'), ('916235064', 'dokker_1027_gao__port_506331', 0.005356989, 332, '355'), ('916235064', '205_1027_gao__port_506062', 0.0066854446, 332, '355'), ('916235064', 'macan_1027_gao__port_506195', 0.0015572828, 332, '355'), ('916235064', 'tourneo_1027_gao__port_506369', 0.006401112, 332, '355'), ('916235064', '108_1027_gao__port_506384', 0.0052642752, 332, '355'), ('916235064', '9_3_1027_gao__port_506071', 0.0008374639, 332, '355'), ('916235064', 'mondeo_1027_gao__port_506116', 0.0014396489, 332, '355'), ('916235064', 'cr_v_1027_gao__port_506164', 0.0016414478, 332, '355'), ('916235064', 'c30_1027_gao__port_506090', 0.0017486482, 332, '355'), ('916235064', 'pulsar_1027_gao__port_506397', 0.0012019476, 332, '355'), ('916235064', 'ibiza_1027_gao__port_506273', 0.003723068, 332, '355'), ('916235064', 'a1_1027_gao__port_506338', 0.0012346336, 332, '355'), ('916235064', 'matrix_1027_gao__port_506140', 0.00070773193, 332, '355'), ('916235064', 'carnival_1027_gao__port_506136', 0.0022812954, 332, '355'), ('916235064', 'xantia_1027_gao__port_506086', 0.0021961364, 332, '355'), ('916235064', 'terrano_1027_gao__port_506083', 0.0020295626, 332, '355'), ('916235064', 'q3_1027_gao__port_506275', 0.0011265541, 332, '355'), ('916235064', 'hr_v_1027_gao__port_506283', 0.0017805516, 332, '355'), ('916235064', 'expert_1027_gao__port_506142', 0.007369358, 332, '355'), ('916235064', 'multivan_1027_gao__port_506383', 0.00650415, 332, '355'), ('916235064', 'venga_1027_gao__port_506380', 0.0008004501, 332, '355'), ('916235064', 'scudo_1027_gao__port_506129', 0.0055921557, 332, '355'), ('916235064', 'laguna_1027_gao__port_506368', 0.00071343547, 332, '355'), ('916235064', 'vel_satis_1027_gao__port_506130', 0.0027262757, 332, '355'), ('916235064', 'b_max_1027_gao__port_506367', 0.0017246609, 332, '355'), ('916235064', 'ignis_1027_gao__port_506292', 0.0043562595, 332, '355'), ('916235064', '159_1027_gao__port_506064', 0.00107823, 332, '355'), ('916235064', 'grande_punto_1027_gao__port_506138', 0.0023634231, 332, '355'), ('916235064', 'logan_1027_gao__port_506167', 0.004397029, 332, '355'), ('916235064', 's_max_1027_gao__port_506223', 0.0012526793, 332, '355'), ('916235064', 'caravelle_1027_gao__port_506351', 0.003029506, 332, '355'), ('916235064', 'adam_1027_gao__port_506079', 0.0010538007, 332, '355'), ('916235064', '406_1027_gao__port_506132', 0.0013575241, 332, '355'), ('916235064', 'q30_1027_gao__port_506293', 0.00097158377, 332, '355'), ('916235064', 'almera_1027_gao__port_506089', 0.0010240278, 332, '355'), ('916235064', 'corsa_1027_gao__port_506095', 0.0025203922, 332, '355'), ('916235064', 'corolla_1027_gao__port_506120', 0.002682061, 332, '355'), ('916235064', 'xc60_1027_gao__port_506388', 0.0018986261, 332, '355'), ('916235064', 'viano_1027_gao__port_506211', 0.0026942661, 332, '355'), ('916235064', 'pro_cee_d_1027_gao__port_506274', 0.00083204353, 332, '355'), ('916235064', 'a3_1027_gao__port_506321', 0.0037380618, 332, '355'), ('916235064', 'v50_1027_gao__port_506150', 0.0007919942, 332, '355'), ('916235064', 'voyager_1027_gao__port_506169', 0.0030524412, 332, '355'), ('916235064', 'corvette_1027_gao__port_506049', 0.0037233331, 332, '355'), ('916235064', 'rio_1027_gao__port_506379', 0.0017740269, 332, '355'), ('916235064', 'jazz_1027_gao__port_506252', 0.00153049, 332, '355'), ('916235064', '200_1027_gao__port_506112', 0.0040874337, 332, '355'), ('916235064', 'tts_1027_gao__port_506199', 0.0011862915, 332, '355'), ('916235064', 'zafira_1027_gao__port_506287', 0.0026952967, 332, '355'), ('916235064', 'asx_1027_gao__port_506266', 0.0011407654, 332, '355'), ('916235064', '607_1027_gao__port_506118', 0.0012529469, 332, '355'), ('916235064', '207_1027_gao__port_506103', 0.0015148912, 332, '355'), ('916235064', 'classe_s_1027_gao__port_506301', 0.0031656828, 332, '355'), ('916235064', 'c6_1027_gao__port_506105', 0.001734882, 332, '355'), ('916235064', 'express_1027_gao__port_506137', 0.016726544, 332, '355'), ('916235064', 'classe_gla_1027_gao__port_506352', 0.0018255792, 332, '355'), ('916235064', 'v60_1027_gao__port_506333', 0.002145976, 332, '355'), ('916235064', 'ka_1027_gao__port_506180', 0.00141526, 332, '355'), ('916235064', 'range_rover_1027_gao__port_506254', 0.0020552275, 332, '355'), ('916235064', 'discovery_1027_gao__port_506375', 0.0022962564, 332, '355'), ('916235064', 'classe_r_1027_gao__port_506270', 0.0013944675, 332, '355'), ('916235064', 'transporter_1027_gao__port_506319', 0.011967846, 332, '355'), ('916235064', 'cee_d_1027_gao__port_506288', 0.0010548627, 332, '355'), ('916235064', 'zoe_1027_gao__port_506244', 0.0020714684, 332, '355'), ('916235064', 'i20_1027_gao__port_506284', 0.0017869968, 332, '355'), ('916235064', 'gtv_1027_gao__port_506059', 0.0057224976, 332, '355'), ('916235064', 's4_avant_1027_gao__port_506261', 0.0027666432, 332, '355'), ('916235064', 'x1_1027_gao__port_506372', 0.0017145572, 332, '355'), ('916235064', 'autres_1027_gao__port_506127', 0.0048251273, 332, '355'), ('916235064', '208_1027_gao__port_506359', 0.0018687285, 332, '355'), ('916235064', 'c8_1027_gao__port_506135', 0.0012579234, 332, '355'), ('916235064', 'astra_1027_gao__port_506215', 0.0012625888, 332, '355'), ('916235064', '2_1027_gao__port_506151', 0.0009245054, 332, '355'), ('916235064', 'doblo_1027_gao__port_506251', 0.0074657435, 332, '355'), ('916235064', '807_1027_gao__port_506152', 0.00072902907, 332, '355'), ('916235064', '206_1027_gao__port_506126', 0.0010386389, 332, '355'), ('916235064', 'a7_1027_gao__port_506373', 0.00069117296, 332, '355'), ('916235064', 'renegade_1027_gao__port_506346', 0.0021416973, 332, '355')]]} begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 7.867813110351562e-06 save missing photos in datou_result : time spend for datou_step_exec : 6.942572832107544 time spend to save output : 1.7644503116607666 total time spend for step 1 : 8.70702314376831 step2:argmax Thu May 15 06:36:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747283780_1979549_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1747283780_1979549_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step Argmax ! calculate argmax for thcl : 355 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True photo_id : 916235064 output[photo_id] : [('916235064', 'c15_1027_gao__port_506055', 0.017712925, 332, '355'), 'temp/1747283780_1979549_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg'] begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 1 insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) first line : ('916235064', '2049863950', '332') ... last line : ('916235064', '2049863950', '332') time used for this insertion : 0.012076139450073242 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 1 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 0.017905235290527344 len list_finale : 1, len picture : 1 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('2', None, '916235064', 'c15_1027_gao__port_506055', None, None, '2049863950', '0.017712925', None)] time used for this insertion : 0.015086650848388672 saving photo_ids in datou_result photo id not in port begin to insert list_values into mtr_datou_result : length of list_values in save_final : 0 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [] time used for this insertion : 5.0067901611328125e-06 save missing photos in datou_result : time spend for datou_step_exec : 0.0002455711364746094 time spend to save output : 0.045479536056518555 total time spend for step 2 : 0.045725107192993164 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : {'916235064': [('916235064', 'c15_1027_gao__port_506055', 0.017712925, 332, '355'), 'temp/1747283780_1979549_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg']} ############################### TEST tfhub2 ################################ TEST TFHUB2 ######################## test with use_multi_inputs=0 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4567 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4567 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4567 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4567 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12835 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12836 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171252784,1171252764,1171252487) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171252487 begin to download photo : 1171252764 begin to download photo : 1171252784 download finish for photo 1171252487 download finish for photo 1171252764 download finish for photo 1171252784 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.20020699501037598 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Thu May 15 06:36:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747283789_1979549_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg': 1171252487, 'temp/1747283789_1979549_1171252764_29d5179a892cc50aadc9d67245534b59.jpg': 1171252764, 'temp/1747283789_1979549_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg': 1171252784} map_photo_id_path_extension : {1171252487: {'path': 'temp/1747283789_1979549_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg', 'extension': 'jpg'}, 1171252764: {'path': 'temp/1747283789_1979549_1171252764_29d5179a892cc50aadc9d67245534b59.jpg', 'extension': 'jpg'}, 1171252784: {'path': 'temp/1747283789_1979549_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3609': 1} we are using the classfication for only one thcl 3609 begin to check gpu status inside check gpu memory 2025-05-15 06:36:34.127516: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-15 06:36:34.129723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 06:36:34.129857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:36:34.129912: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:36:34.153336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 06:36:34.153481: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 06:36:34.199464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 06:36:34.205806: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 06:36:34.269262: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:36:34.270649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 06:36:34.271613: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-15 06:36:34.399098: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-15 06:36:34.401160: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f78e8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-15 06:36:34.401192: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-15 06:36:34.406451: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x64e3e880 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-15 06:36:34.406476: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-15 06:36:34.407515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 06:36:34.407630: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:36:34.407655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 06:36:34.407736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 06:36:34.407775: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 06:36:34.407810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 06:36:34.407850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 06:36:34.407888: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 06:36:34.408898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 06:36:34.409307: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 06:36:34.409362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-15 06:36:34.409375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-15 06:36:34.409384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-15 06:36:34.410752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3154 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3609 To do loadFromThcl(), then load ParamDescType : thcl3609 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3609) thcls : [{'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'}] thcl {'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'} Update svm_hashtag_type_desc : 5832 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5832) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5832, 'tfhub_19_06_2023', 1280, 1280, 'tfhub_19_06_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 6, 19, 12, 55, 22), datetime.datetime(2023, 6, 19, 12, 55, 22)) model_name : tfhub_19_06_2023 model_param file didn't exist model_name : tfhub_19_06_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/tfhub_19_06_2023 /data/models_weight/tfhub_19_06_2023/Confusion_Matrix.png size_local : 57753 size in s3 : 57753 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_jrm.jpg size_local : 79724 size in s3 : 79724 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcm.jpg size_local : 83556 size in s3 : 83556 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcnc.jpg size_local : 74107 size in s3 : 74107 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pehd.jpg size_local : 72705 size in s3 : 72705 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_tapis_vide.jpg size_local : 70874 size in s3 : 70874 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 checkpoint already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216488 size in s3 : 216488 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:21 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_weights.h5 size_local : 16499144 size in s3 : 16499144 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:15 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3139, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 77, in create_tfhub_model hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module"), File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171252487, 1171252764, 1171252784] map_info['map_portfolio_photo'] : {} final : True mtd_id 4567 list_pids : [1171252487, 1171252764, 1171252784] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4567', None, '1171252487', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252764', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252784', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.0166018009185791 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 12.772142171859741 time spend to save output : 0.026662826538085938 total time spend for step 0 : 12.798804998397827 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171252784': [(1171252784, 'jrm', 0.9677492, 4674, '3609'), 'temp/1687511175_1882837_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg'], '1171252764': [(1171252764, 'jrm', 0.9853587, 4674, '3609'), 'temp/1687511175_1882837_1171252764_29d5179a892cc50aadc9d67245534b59.jpg'], '1171252487': [(1171252487, 'jrm', 0.9262757, 4674, '3609'), 'temp/1687511175_1882837_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg']} got : None ######################## test with use_multi_inputs=1 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4621 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4621 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4621 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4621 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12927 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12928 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171291875,1171275372,1171275314) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171275314 begin to download photo : 1171275372 begin to download photo : 1171291875 download finish for photo 1171275372 download finish for photo 1171275314 download finish for photo 1171291875 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.18609142303466797 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Thu May 15 06:36:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747283802_1979549_1171275372_76d81364ff7df843bff095f45c07ba35.jpg': 1171275372, 'temp/1747283802_1979549_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg': 1171275314, 'temp/1747283802_1979549_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg': 1171291875} map_photo_id_path_extension : {1171275372: {'path': 'temp/1747283802_1979549_1171275372_76d81364ff7df843bff095f45c07ba35.jpg', 'extension': 'jpg'}, 1171275314: {'path': 'temp/1747283802_1979549_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg', 'extension': 'jpg'}, 1171291875: {'path': 'temp/1747283802_1979549_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3655': 1} we are using the classfication for only one thcl 3655 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 24 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 24 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 24 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 24 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 24 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 24 wait 20 seconds l 3637 free memory gpu now : 24 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3655 To do loadFromThcl(), then load ParamDescType : thcl3655 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3655) thcls : [{'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'}] thcl {'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'} Update svm_hashtag_type_desc : 5862 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5862) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5862, 'tfhub_18_7_2023', 1280, 1280, 'tfhub_18_7_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 7, 18, 22, 46, 29), datetime.datetime(2023, 7, 18, 22, 46, 29)) model_name : tfhub_18_7_2023 model_param file didn't exist model_name : tfhub_18_7_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/tfhub_18_7_2023 /data/models_weight/tfhub_18_7_2023/Confusion_Matrix.png size_local : 54360 size in s3 : 54360 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:28 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_jrm.jpg size_local : 72583 size in s3 : 72583 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcm.jpg size_local : 81681 size in s3 : 81681 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcnc.jpg size_local : 79510 size in s3 : 79510 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pehd.jpg size_local : 59936 size in s3 : 59936 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_tapis_vide.jpg size_local : 78974 size in s3 : 78974 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 checkpoint already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216529 size in s3 : 216529 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279748 size in s3 : 32279748 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_weights.h5 size_local : 16500868 size in s3 : 16500868 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:18 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3139, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 62, in create_tfhub_model fe_layer = hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module", File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171275372, 1171275314, 1171291875] map_info['map_portfolio_photo'] : {} final : True mtd_id 4621 list_pids : [1171275372, 1171275314, 1171291875] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4621', None, '1171275372', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171275314', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171291875', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.03356790542602539 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 135.93277835845947 time spend to save output : 0.03549933433532715 total time spend for step 0 : 135.9682776927948 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171291875': [(1171291875, 'tapis_vide', 0.97062814, 4723, '3655'), 'temp/1691745841_1143057_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg'], '1171275372': [(1171275372, 'tapis_vide', 0.9674145, 4723, '3655'), 'temp/1691745841_1143057_1171275372_76d81364ff7df843bff095f45c07ba35.jpg'], '1171275314': [(1171275314, 'tapis_vide', 0.96509415, 4723, '3655'), 'temp/1691745841_1143057_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg']} got : None ERROR tfhub2 FAILED ############################### TEST ordonner ################################ To do loadFromThcl(), then load ParamDescType : thcl358 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (358) thcls : [{'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3392 ['FirstUploadExperveo_vignette__port_505674', 'CAR_EXTERIEUR_Roue__port_503398', 'FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486', 'FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465', 'CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198', 'CAR_EXTERIEUR_Face_avant_axe_droit__port_504451', 'CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235', 'FirstUploadExperveo_vin__port_505675', 'CAR_EXTERIEUR_cote_droite__port_504108', 'CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565', 'FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201', 'cartegrise_orientation__port_505064', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217', 'CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531', 'CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218', 'CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214', 'CAR_EXTERIEUR_Angle_avant_droit__port_504087', 'FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484', 'CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563', 'CAR_EXTERIEUR_Angle_arriere_droit__port_504160', 'CAR_EXTERIEUR_arriere__port_504184', 'CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562', 'INTERIEUR_Compteur_kilometrique__port_503644', 'CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494', 'CAR_EXTERIEUR_Angle_arriere_gauche__port_504170', 'CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226', 'CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202', 'CAR_EXTERIEUR_moteur__port_503704', 'FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487', 'CAR_INTERIEUR_siege_arriere_class_1__port_506551', 'CAR_EXTERIEUR_avant__port_504146', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215', 'CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225', 'CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564', 'FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'CAR_INTERIEUR_coffre__port_503412', 'FirstUploadExperveo_rouetranche__port_505677', 'UploadPhotoImmatBest_class_1__port_505051', 'CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532', 'CAR_EXTERIEUR_angle_avant_gauche__port_504098', 'CAR_EXTERIEUR_face_avant_axe_gauche__port_504236', 'CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540', 'CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233', 'CAR_EXTERIEUR_roue_de_secour__port_503763', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199', 'CAR_EXTERIEUR_cote_gauche__port_504017', 'CAR_INTERIEUR_avant_volant_class_1__port_506503', 'CAR_INTERIEUR_avant_volant_class_2__port_506504', 'CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'] 51 SELECT hashtag_id,hashtag FROM MTRBack.hashtags where hashtag in ('FirstUploadExperveo_vignette__port_505674','CAR_EXTERIEUR_Roue__port_503398','FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486','FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485','CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465','CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198','CAR_EXTERIEUR_Face_avant_axe_droit__port_504451','CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235','FirstUploadExperveo_vin__port_505675','CAR_EXTERIEUR_cote_droite__port_504108','CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565','FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483','CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201','cartegrise_orientation__port_505064','CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217','CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531','CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218','CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214','CAR_EXTERIEUR_Angle_avant_droit__port_504087','FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484','CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563','CAR_EXTERIEUR_Angle_arriere_droit__port_504160','CAR_EXTERIEUR_arriere__port_504184','CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562','INTERIEUR_Compteur_kilometrique__port_503644','CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494','CAR_EXTERIEUR_Angle_arriere_gauche__port_504170','CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226','CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202','CAR_EXTERIEUR_moteur__port_503704','FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487','CAR_INTERIEUR_siege_arriere_class_1__port_506551','CAR_EXTERIEUR_avant__port_504146','CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215','CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225','CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564','FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482','CAR_INTERIEUR_coffre__port_503412','FirstUploadExperveo_rouetranche__port_505677','UploadPhotoImmatBest_class_1__port_505051','CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532','CAR_EXTERIEUR_angle_avant_gauche__port_504098','CAR_EXTERIEUR_face_avant_axe_gauche__port_504236','CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540','CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233','CAR_EXTERIEUR_roue_de_secour__port_503763','CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199','CAR_EXTERIEUR_cote_gauche__port_504017','CAR_INTERIEUR_avant_volant_class_1__port_506503','CAR_INTERIEUR_avant_volant_class_2__port_506504','CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'); 51 dict_keys(['cartegrise_orientation__port_505064', 'car_exterieur_angle_arriere_droit_axe_arriere__port_504217', 'car_exterieur_angle_arriere_droit_axe_droit__port_504215', 'car_exterieur_angle_arriere_droit__port_504160', 'car_exterieur_angle_arriere_gauche_axe_arriere__port_504201', 'car_exterieur_angle_arriere_gauche_axe_gauche__port_504199', 'car_exterieur_angle_arriere_gauche__port_504170', 'car_exterieur_angle_avant_droit_axe_arriere__port_504226', 'car_exterieur_angle_avant_droit_axe_droit__port_504225', 'car_exterieur_angle_avant_droit__port_504087', 'car_exterieur_angle_avant_gauche_axe_avant__port_504235', 'car_exterieur_angle_avant_gauche_axe_gauche__port_504234', 'car_exterieur_angle_avant_gauche__port_504098', 'car_exterieur_arriere__port_504184', 'car_exterieur_avant__port_504146', 'car_exterieur_cote_droite__port_504108', 'car_exterieur_cote_droit_axe_arriere__port_504214', 'car_exterieur_cote_droit_axe_avant__port_504465', 'car_exterieur_cote_gauche_axe_arriere__port_504198', 'car_exterieur_cote_gauche_axe_avant__port_504233', 'car_exterieur_cote_gauche__port_504017', 'car_exterieur_face_arriere_axe_droit__port_504218', 'car_exterieur_face_arriere_axe_gauche__port_504202', 'car_exterieur_face_avant_axe_droit__port_504451', 'car_exterieur_face_avant_axe_gauche__port_504236', 'car_exterieur_moteur__port_503704', 'car_exterieur_roue_de_secour__port_503763', 'car_exterieur_roue__port_503398', 'car_interieur_avant_volant_class_1__port_506503', 'car_interieur_avant_volant_class_2__port_506504', 'car_interieur_avant_volant_class_6_boutonrond__port_506562', 'car_interieur_avant_volant_class_6_class_2__port_506563', 'car_interieur_avant_volant_class_6_ecrangrosplan__port_506564', 'car_interieur_avant_volant_class_6_levierdevitesse__port_506565', 'car_interieur_avant_vue-arriere_class_1__port_506531', 'car_interieur_avant_vue-arriere_class_2__port_506532', 'car_interieur_avant_vue_droite_habitacle_class_1__port_506540', 'car_interieur_avant_vue_gauche_habitacle_class_1__port_506494', 'car_interieur_coffre__port_503412', 'car_interieur_siege_arriere_class_1__port_506551', 'firstuploadexperveo_carrosseriegrosplan_carrosserie__port_506483', 'firstuploadexperveo_carrosseriegrosplan_class_6__port_506487', 'firstuploadexperveo_carrosseriegrosplan_morceauderoue__port_506484', 'firstuploadexperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'firstuploadexperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'firstuploadexperveo_carrosseriegrosplan_vindanslamoquette__port_506486', 'firstuploadexperveo_rouetranche__port_505677', 'firstuploadexperveo_vignette__port_505674', 'firstuploadexperveo_vin__port_505675', 'interieur_compteur_kilometrique__port_503644', 'uploadphotoimmatbest_class_1__port_505051']) select photo_hashtag_type from MTRDatou.classification_theme where id = 358 thcl : 358 photo_hashtag_type : 337 SELECT phi.hashtag_id , phi.photo_id FROM MTRBack.photo_hashtag_ids phi, MTRUser.mtr_portfolio_photos mp where phi.type = 337 and phi.photo_id = mp.mtr_photo_id and mp.mtr_portfolio_id =510365; {510365: [(917973295, 1), (917973297, 1), (917973302, 1), (917973293, 7), (917973296, 11), (917973300, 11), (917973286, 13), (917973289, 13), (917973301, 24), (917973285, 29), (917973290, 29), (917973299, 29), (917973304, 35), (917973287, 36), (917973298, 36), (917973305, 36), (917973292, 37), (917973291, 41), (917973303, 41), (917973294, 42), (917973288, 46)]} ############################### TEST rotate ################################ test rotate only Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=230 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=230 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 230 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=230 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.1260836124420166 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:rotate Thu May 15 06:38:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step_rotate ! We are in a linear step without datou_depend ! rotate photos of 90,180,270 degres batch 1 select photo_id, hashtag_id, `type`, x0, x1, y0, y1, score, id from MTRPhoto.crop_hashtag_ids where photo_id in ( 917849322) and `type` in (0) Loaded 0 chid ids of type : 0 SELECT crop_hashtag_id, points FROM MTRPhoto.crop_polygon_points WHERE crop_hashtag_id in () map_chi : {} photo_id in download_rotate_and_save : 917849322 list_chi_loc : 0 Use all angle ! Rotation of photo 917849322 of 90 degree temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 90 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] 90 [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] shrink_image : False image_rotate : image_path : temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 180 degree temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 180 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] 180 [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] shrink_image : False image_rotate : image_path : temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 270 degree temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 270 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] 270 [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] shrink_image : False image_rotate : image_path : temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg image_rotate.mode : RGB About to upload 3 photos upload in portfolio : 551782 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1747283940_1979549 we have uploaded 3 photos in the portfolio 551782 time of upload the photos Elapsed time : 1.4325814247131348 map_filename_photo_id : 3 map_filename_photo_id : {'temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg': 1358431727, 'temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg': 1358431728, 'temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg': 1358431729} Len new_chis : 3 Len list_new_chi_with_photo_id : 0 of type : 0 list_new_chi_with_photo_id : [] After datou_step_exec type output : time spend for datou_step_exec : 1.6675899028778076 time spend to save output : 8.463859558105469e-05 total time spend for step 1 : 1.6676745414733887 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : rotate we use saveGeneral [917849322] map_info['map_portfolio_photo'] : {} final : True mtd_id 230 list_pids : [917849322] Looping around the photos to save general results len do output : 3 /1358431727Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358431728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358431729Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('230', None, None, None, None, None, None, None, None) ('230', None, '917849322', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('230', None, '1358431727', 'None', None, None, None, None, None), ('230', None, '1358431728', 'None', None, None, None, None, None), ('230', None, '1358431729', 'None', None, None, None, None, None), ('230', None, '917849322', None, None, None, None, None, None)] time used for this insertion : 0.16558170318603516 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {1358431727: ['917849322', 'temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg', []], 1358431728: ['917849322', 'temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg', []], 1358431729: ['917849322', 'temp/1747283939_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg', []]} test rotate only is a success ! test rotate conditionnel Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=233 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=233 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 233 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=233 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax, rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.2751152515411377 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 3 step1:thcl Thu May 15 06:39:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747283941_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1747283941_1979549_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'500': 1} we are using the classfication for only one thcl 500 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (2448, 3264, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.00017213821411132812 time to convert the images to numpy array : 1.8658390045166016 total time to convert the images to numpy array : 1.8665196895599365 list photo_ids error: [] list photo_ids correct : [917849322] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 500 To do loadFromThcl(), then load ParamDescType : thcl500 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (500) thcls : [{'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'}] thcl {'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'} Update svm_hashtag_type_desc : 3517 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) To loadFromThcl() : net_3517 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 24 wait 20 seconds l 3637 free memory gpu now : 24 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) param : , param.caffemodel : orientation_carte_grise_all_2 None mean_file_type : mean_file_path : prototxt_file_path : model : orientation_carte_grise_all_2 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : orientation_carte_grise_all_2 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/orientation_carte_grise_all_2 /data/models_weight/orientation_carte_grise_all_2/caffemodel size_local : 537110520 size in s3 : 537110520 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:17 caffemodel already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_fc.prototxt size_local : 1130 size in s3 : 1130 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt size_local : 5653 size in s3 : 5653 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:31 mean.npy already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/synset_words.txt size_local : 159 size in s3 : 159 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt caffemodel_filename : /data/models_weight/orientation_carte_grise_all_2/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 24 wait 20 seconds WARNING: Logging before InitGoogleLogging() is written to STDERR F0515 06:39:54.215926 1979549 syncedmem.cpp:78] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 31.87user 26.55system 4:28.18elapsed 21%CPU (0avgtext+0avgdata 3532796maxresident)k 7710064inputs+21936outputs (17306major+3010010minor)pagefaults 0swaps