python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5282 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.12019681930541992 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu Apr 3 06:35:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5282 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-04-03 06:35:31.181998: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-04-03 06:35:31.215223: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-04-03 06:35:31.217786: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb9b0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-04-03 06:35:31.217860: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-04-03 06:35:31.222685: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-04-03 06:35:31.466974: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f685740 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-04-03 06:35:31.467035: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-04-03 06:35:31.467931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-03 06:35:31.468283: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:35:31.470355: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:35:31.486695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-03 06:35:31.487132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-03 06:35:31.515812: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-03 06:35:31.519805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-03 06:35:31.573741: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:35:31.575259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-03 06:35:31.575706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:35:31.576451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-04-03 06:35:31.576469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-04-03 06:35:31.576480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-04-03 06:35:31.578443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4823 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-04-03 06:35:32.208894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-03 06:35:32.208976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:35:32.208994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:35:32.209009: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-03 06:35:32.209025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-03 06:35:32.209040: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-03 06:35:32.209065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-03 06:35:32.209081: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:35:32.210083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-03 06:35:32.211259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-03 06:35:32.211299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:35:32.211318: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:35:32.211334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-03 06:35:32.211351: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-03 06:35:32.211367: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-03 06:35:32.211383: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-03 06:35:32.211399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:35:32.212384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-03 06:35:32.212418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-04-03 06:35:32.212427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-04-03 06:35:32.212435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-04-03 06:35:32.213471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4823 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-04-03 06:35:41.006596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:35:41.221317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:35:43.172542: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (480, 640, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 1795215 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 24 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4905 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0006392002105712891 nb_pixel_total : 15574 time to create 1 rle with old method : 0.021159648895263672 length of segment : 255 time for calcul the mask position with numpy : 0.0030829906463623047 nb_pixel_total : 145598 time to create 1 rle with old method : 0.16536641120910645 length of segment : 371 time for calcul the mask position with numpy : 0.00024056434631347656 nb_pixel_total : 14235 time to create 1 rle with old method : 0.01672506332397461 length of segment : 151 time for calcul the mask position with numpy : 0.00011706352233886719 nb_pixel_total : 5816 time to create 1 rle with old method : 0.007380247116088867 length of segment : 49 time for calcul the mask position with numpy : 6.222724914550781e-05 nb_pixel_total : 1701 time to create 1 rle with old method : 0.0022614002227783203 length of segment : 38 time spent for convertir_results : 1.03729248046875 time spend for datou_step_exec : 21.23856496810913 time spend to save output : 4.673004150390625e-05 total time spend for step 1 : 21.238611698150635 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 3327 chid ids of type : 445 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 609 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.017407655715942383 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'957285035': [[(957285035, 492601069, 445, 0, 186, 22, 282, 0.99571794, [(139, 26, 9), (135, 27, 16), (133, 28, 19), (131, 29, 23), (127, 30, 28), (121, 31, 35), (9, 32, 5), (117, 32, 40), (7, 33, 42), (111, 33, 47), (6, 34, 63), (105, 34, 54), (5, 35, 87), (99, 35, 60), (4, 36, 156), (3, 37, 157), (3, 38, 157), (3, 39, 157), (2, 40, 158), (2, 41, 158), (2, 42, 158), (2, 43, 158), (2, 44, 158), (1, 45, 159), (1, 46, 159), (1, 47, 159), (1, 48, 158), (1, 49, 158), (1, 50, 158), (1, 51, 158), (1, 52, 157), (1, 53, 157), (1, 54, 156), (1, 55, 153), (1, 56, 150), (1, 57, 147), (1, 58, 142), (1, 59, 138), (1, 60, 134), (1, 61, 131), (1, 62, 128), (1, 63, 127), (1, 64, 125), (1, 65, 124), (1, 66, 122), (1, 67, 120), (1, 68, 119), (1, 69, 118), (1, 70, 116), (1, 71, 115), (1, 72, 114), (1, 73, 113), (1, 74, 112), (1, 75, 111), (1, 76, 110), (1, 77, 109), (1, 78, 108), (1, 79, 107), (1, 80, 106), (1, 81, 106), (2, 82, 104), (2, 83, 103), (2, 84, 103), (2, 85, 102), (2, 86, 102), (2, 87, 101), (2, 88, 100), (2, 89, 99), (2, 90, 98), (2, 91, 98), (2, 92, 97), (2, 93, 96), (2, 94, 95), (2, 95, 93), (2, 96, 91), (2, 97, 90), (2, 98, 89), (2, 99, 87), (2, 100, 86), (2, 101, 86), (2, 102, 85), (2, 103, 84), (2, 104, 83), (2, 105, 83), (2, 106, 82), (2, 107, 81), (2, 108, 80), (2, 109, 80), (2, 110, 79), (2, 111, 78), (2, 112, 77), (2, 113, 76), (1, 114, 76), (1, 115, 75), (1, 116, 74), (1, 117, 73), (1, 118, 72), (1, 119, 71), (1, 120, 71), (1, 121, 70), (1, 122, 69), (1, 123, 69), (1, 124, 68), (1, 125, 68), (1, 126, 67), (1, 127, 67), (1, 128, 66), (1, 129, 66), (1, 130, 66), (1, 131, 65), (1, 132, 65), (1, 133, 64), (1, 134, 63), (1, 135, 63), (1, 136, 62), (1, 137, 61), (1, 138, 60), (1, 139, 60), (1, 140, 59), (1, 141, 58), (1, 142, 58), (1, 143, 57), (1, 144, 56), (1, 145, 56), (1, 146, 55), (1, 147, 55), (1, 148, 54), (1, 149, 53), (1, 150, 52), (1, 151, 52), (1, 152, 51), (1, 153, 50), (1, 154, 49), (1, 155, 48), (1, 156, 47), (1, 157, 46), (1, 158, 45), (1, 159, 45), (1, 160, 44), (1, 161, 43), (1, 162, 42), (1, 163, 41), (1, 164, 41), (1, 165, 40), (1, 166, 40), (1, 167, 39), (1, 168, 38), (1, 169, 37), (1, 170, 36), (1, 171, 35), (1, 172, 34), (1, 173, 34), (1, 174, 33), (1, 175, 32), (1, 176, 32), (1, 177, 32), (1, 178, 32), (1, 179, 31), (1, 180, 31), (1, 181, 31), (1, 182, 31), (1, 183, 30), (1, 184, 30), (1, 185, 30), (1, 186, 29), (1, 187, 29), (1, 188, 28), (1, 189, 28), (1, 190, 28), (1, 191, 27), (1, 192, 27), (1, 193, 26), (1, 194, 26), (1, 195, 26), (1, 196, 26), (1, 197, 26), (1, 198, 26), (1, 199, 26), (1, 200, 25), (1, 201, 25), (1, 202, 25), (1, 203, 25), (1, 204, 25), (1, 205, 25), (1, 206, 25), (1, 207, 25), (1, 208, 25), (1, 209, 25), (1, 210, 25), (1, 211, 25), (1, 212, 25), (1, 213, 25), (1, 214, 25), (1, 215, 25), (1, 216, 25), (1, 217, 25), (1, 218, 25), (1, 219, 25), (1, 220, 25), (1, 221, 24), (1, 222, 24), (1, 223, 24), (1, 224, 25), (1, 225, 25), (1, 226, 25), (1, 227, 25), (1, 228, 25), (2, 229, 24), (2, 230, 24), (2, 231, 24), (2, 232, 24), (2, 233, 23), (2, 234, 23), (2, 235, 23), (2, 236, 23), (2, 237, 23), (2, 238, 23), (2, 239, 23), (2, 240, 23), (2, 241, 23), (2, 242, 23), (2, 243, 23), (2, 244, 23), (2, 245, 23), (2, 246, 23), (2, 247, 24), (2, 248, 24), (2, 249, 24), (2, 250, 24), (2, 251, 24), (2, 252, 24), (2, 253, 24), (2, 254, 23), (2, 255, 23), (2, 256, 23), (2, 257, 23), (2, 258, 23), (2, 259, 23), (2, 260, 23), (2, 261, 23), (3, 262, 22), (3, 263, 22), (3, 264, 22), (3, 265, 23), (4, 266, 22), (4, 267, 22), (5, 268, 20), (5, 269, 20), (6, 270, 19), (7, 271, 18), (8, 272, 16), (8, 273, 16), (9, 274, 14), (10, 275, 11), (14, 276, 4)], ['10,275,5,269,2,261,2,229,1,228,1,114,2,113,2,82,1,81,1,45,3,37,9,32,14,33,48,33,49,34,68,34,69,35,91,35,92,36,116,33,121,31,130,30,139,26,147,26,153,29,159,36,159,47,156,54,129,61,117,69,106,80,103,86,96,94,89,98,81,109,71,119,65,132,60,138,55,147,41,163,40,166,32,175,31,182,26,193,25,200,25,232,24,233,24,264,25,267,22,274']), (957285035, 492601069, 445, 31, 591, 25, 418, 0.99287707, [(303, 37, 43), (267, 38, 102), (250, 39, 136), (231, 40, 162), (197, 41, 202), (185, 42, 232), (177, 43, 250), (171, 44, 262), (166, 45, 273), (162, 46, 282), (160, 47, 289), (159, 48, 295), (157, 49, 303), (156, 50, 308), (154, 51, 314), (153, 52, 318), (151, 53, 324), (149, 54, 330), (148, 55, 334), (146, 56, 338), (144, 57, 341), (141, 58, 345), (139, 59, 349), (136, 60, 352), (135, 61, 354), (133, 62, 357), (131, 63, 360), (129, 64, 362), (127, 65, 365), (125, 66, 368), (123, 67, 371), (121, 68, 374), (119, 69, 377), (118, 70, 379), (116, 71, 382), (115, 72, 384), (114, 73, 386), (113, 74, 388), (112, 75, 391), (111, 76, 393), (110, 77, 395), (110, 78, 396), (109, 79, 398), (109, 80, 399), (108, 81, 401), (107, 82, 403), (107, 83, 404), (106, 84, 406), (106, 85, 407), (105, 86, 409), (105, 87, 410), (104, 88, 411), (103, 89, 413), (102, 90, 415), (101, 91, 417), (100, 92, 420), (99, 93, 422), (97, 94, 426), (96, 95, 428), (95, 96, 430), (94, 97, 432), (93, 98, 434), (91, 99, 437), (90, 100, 439), (89, 101, 441), (88, 102, 443), (88, 103, 443), (88, 104, 444), (88, 105, 445), (88, 106, 446), (88, 107, 446), (88, 108, 447), (88, 109, 448), (88, 110, 449), (88, 111, 451), (88, 112, 452), (87, 113, 454), (87, 114, 455), (87, 115, 457), (87, 116, 457), (86, 117, 459), (86, 118, 460), (85, 119, 461), (84, 120, 463), (84, 121, 463), (83, 122, 465), (82, 123, 466), (82, 124, 467), (81, 125, 468), (80, 126, 470), (79, 127, 471), (78, 128, 473), (77, 129, 474), (76, 130, 475), (74, 131, 478), (73, 132, 479), (71, 133, 482), (70, 134, 483), (68, 135, 486), (67, 136, 487), (65, 137, 490), (64, 138, 491), (62, 139, 494), (61, 140, 496), (60, 141, 497), (60, 142, 498), (59, 143, 500), (59, 144, 501), (58, 145, 503), (58, 146, 503), (58, 147, 504), (57, 148, 506), (57, 149, 507), (56, 150, 508), (56, 151, 509), (56, 152, 510), (55, 153, 511), (55, 154, 512), (55, 155, 512), (54, 156, 514), (54, 157, 514), (54, 158, 514), (53, 159, 516), (53, 160, 516), (52, 161, 517), (52, 162, 517), (51, 163, 518), (51, 164, 518), (51, 165, 518), (50, 166, 519), (50, 167, 519), (49, 168, 520), (49, 169, 520), (48, 170, 522), (48, 171, 522), (47, 172, 523), (47, 173, 522), (46, 174, 523), (46, 175, 523), (46, 176, 523), (45, 177, 524), (45, 178, 524), (44, 179, 525), (44, 180, 524), (43, 181, 525), (43, 182, 525), (42, 183, 526), (42, 184, 526), (41, 185, 527), (41, 186, 526), (40, 187, 527), (40, 188, 526), (39, 189, 526), (39, 190, 525), (38, 191, 526), (38, 192, 525), (38, 193, 523), (37, 194, 523), (37, 195, 522), (37, 196, 521), (36, 197, 521), (36, 198, 521), (36, 199, 520), (35, 200, 520), (35, 201, 520), (35, 202, 519), (35, 203, 519), (35, 204, 519), (35, 205, 518), (35, 206, 518), (35, 207, 518), (34, 208, 518), (34, 209, 518), (34, 210, 518), (34, 211, 517), (34, 212, 517), (34, 213, 517), (34, 214, 516), (34, 215, 516), (34, 216, 515), (34, 217, 514), (34, 218, 514), (34, 219, 513), (34, 220, 512), (34, 221, 511), (34, 222, 510), (33, 223, 509), (33, 224, 508), (33, 225, 506), (33, 226, 505), (33, 227, 504), (33, 228, 503), (33, 229, 502), (33, 230, 501), (33, 231, 500), (33, 232, 499), (33, 233, 499), (33, 234, 498), (33, 235, 497), (33, 236, 496), (33, 237, 495), (33, 238, 494), (33, 239, 493), (33, 240, 491), (33, 241, 490), (33, 242, 489), (33, 243, 488), (33, 244, 486), (33, 245, 485), (33, 246, 483), (33, 247, 482), (33, 248, 480), (33, 249, 479), (33, 250, 477), (33, 251, 476), (33, 252, 474), (33, 253, 472), (33, 254, 471), (33, 255, 469), (33, 256, 468), (33, 257, 466), (33, 258, 465), (33, 259, 463), (33, 260, 462), (33, 261, 461), (33, 262, 459), (33, 263, 458), (33, 264, 456), (33, 265, 455), (33, 266, 453), (33, 267, 451), (33, 268, 450), (33, 269, 448), (33, 270, 447), (33, 271, 446), (33, 272, 444), (33, 273, 443), (33, 274, 442), (33, 275, 440), (33, 276, 439), (33, 277, 437), (33, 278, 436), (34, 279, 433), (34, 280, 432), (34, 281, 430), (34, 282, 429), (34, 283, 427), (34, 284, 425), (34, 285, 424), (35, 286, 422), (35, 287, 420), (35, 288, 419), (35, 289, 418), (36, 290, 415), (36, 291, 414), (36, 292, 412), (36, 293, 411), (37, 294, 408), (37, 295, 407), (37, 296, 405), (38, 297, 403), (38, 298, 401), (39, 299, 399), (40, 300, 396), (41, 301, 394), (42, 302, 392), (43, 303, 389), (44, 304, 386), (46, 305, 383), (47, 306, 380), (48, 307, 378), (49, 308, 375), (49, 309, 373), (50, 310, 370), (51, 311, 368), (52, 312, 366), (53, 313, 364), (54, 314, 362), (55, 315, 361), (57, 316, 358), (59, 317, 355), (61, 318, 353), (64, 319, 349), (67, 320, 345), (70, 321, 342), (72, 322, 339), (75, 323, 336), (77, 324, 333), (79, 325, 331), (81, 326, 328), (83, 327, 326), (85, 328, 323), (87, 329, 321), (90, 330, 318), (93, 331, 314), (96, 332, 311), (99, 333, 307), (103, 334, 303), (106, 335, 299), (108, 336, 297), (111, 337, 293), (113, 338, 291), (115, 339, 288), (116, 340, 286), (118, 341, 284), (120, 342, 281), (122, 343, 278), (125, 344, 274), (127, 345, 271), (130, 346, 268), (132, 347, 265), (135, 348, 262), (138, 349, 258), (141, 350, 255), (143, 351, 252), (146, 352, 249), (147, 353, 247), (149, 354, 245), (151, 355, 242), (153, 356, 240), (155, 357, 238), (157, 358, 236), (159, 359, 233), (162, 360, 230), (164, 361, 228), (166, 362, 226), (168, 363, 224), (170, 364, 221), (171, 365, 220), (172, 366, 219), (174, 367, 217), (175, 368, 216), (176, 369, 214), (177, 370, 213), (179, 371, 211), (180, 372, 209), (183, 373, 206), (186, 374, 203), (189, 375, 199), (192, 376, 196), (194, 377, 193), (197, 378, 190), (199, 379, 187), (202, 380, 184), (204, 381, 181), (205, 382, 180), (207, 383, 177), (209, 384, 175), (210, 385, 173), (212, 386, 170), (214, 387, 167), (216, 388, 164), (219, 389, 160), (221, 390, 157), (225, 391, 152), (229, 392, 148), (232, 393, 144), (235, 394, 140), (238, 395, 136), (241, 396, 133), (244, 397, 129), (249, 398, 123), (253, 399, 118), (259, 400, 110), (267, 401, 101), (278, 402, 88), (290, 403, 74), (302, 404, 60), (309, 405, 47), (315, 406, 35), (320, 407, 25)], ['344,407,320,407,301,403,278,402,248,397,221,390,201,379,180,372,169,363,110,336,87,329,55,315,38,298,33,278,33,223,35,200,51,165,60,141,75,131,87,116,88,102,105,87,116,71,147,56,162,46,197,41,230,41,267,38,345,37,416,42,467,51,483,56,490,64,543,115,554,138,568,159,567,185,554,200,547,218,527,237,481,268,452,289,420,309,411,320,403,338,392,355,390,368,383,384,373,396,361,404']), (957285035, 492601069, 445, 484, 636, 22, 174, 0.96813005, [(541, 24, 18), (614, 24, 19), (534, 25, 38), (595, 25, 39), (528, 26, 106), (523, 27, 111), (520, 28, 114), (518, 29, 117), (517, 30, 118), (515, 31, 120), (514, 32, 121), (512, 33, 123), (511, 34, 124), (510, 35, 125), (509, 36, 126), (507, 37, 128), (505, 38, 130), (503, 39, 132), (502, 40, 133), (501, 41, 134), (500, 42, 135), (499, 43, 136), (497, 44, 138), (496, 45, 139), (496, 46, 139), (495, 47, 140), (494, 48, 141), (494, 49, 141), (493, 50, 142), (492, 51, 143), (491, 52, 145), (491, 53, 145), (491, 54, 145), (490, 55, 146), (490, 56, 146), (490, 57, 146), (490, 58, 146), (490, 59, 146), (491, 60, 145), (491, 61, 145), (491, 62, 145), (492, 63, 144), (493, 64, 143), (494, 65, 142), (495, 66, 141), (496, 67, 140), (497, 68, 139), (498, 69, 138), (499, 70, 137), (500, 71, 136), (502, 72, 134), (503, 73, 133), (504, 74, 132), (505, 75, 131), (506, 76, 130), (507, 77, 129), (508, 78, 128), (509, 79, 127), (510, 80, 126), (512, 81, 124), (513, 82, 123), (514, 83, 122), (515, 84, 121), (516, 85, 120), (516, 86, 120), (517, 87, 119), (519, 88, 117), (520, 89, 116), (521, 90, 115), (522, 91, 114), (522, 92, 114), (523, 93, 113), (524, 94, 112), (525, 95, 111), (526, 96, 110), (527, 97, 109), (529, 98, 107), (531, 99, 105), (532, 100, 104), (533, 101, 103), (534, 102, 102), (535, 103, 101), (536, 104, 100), (538, 105, 98), (539, 106, 97), (541, 107, 95), (544, 108, 92), (546, 109, 90), (548, 110, 88), (549, 111, 87), (550, 112, 85), (552, 113, 83), (553, 114, 82), (554, 115, 81), (555, 116, 80), (556, 117, 79), (557, 118, 78), (558, 119, 77), (559, 120, 76), (560, 121, 75), (560, 122, 75), (560, 123, 75), (561, 124, 74), (561, 125, 74), (562, 126, 73), (563, 127, 72), (563, 128, 72), (564, 129, 71), (564, 130, 70), (564, 131, 70), (565, 132, 69), (565, 133, 69), (565, 134, 68), (565, 135, 67), (565, 136, 66), (566, 137, 65), (566, 138, 64), (566, 139, 63), (566, 140, 61), (566, 141, 59), (566, 142, 57), (566, 143, 56), (566, 144, 55), (566, 145, 55), (567, 146, 53), (567, 147, 52), (567, 148, 52), (568, 149, 50), (568, 150, 49), (568, 151, 48), (569, 152, 46), (569, 153, 45), (570, 154, 44), (570, 155, 43), (570, 156, 42), (571, 157, 40), (571, 158, 39), (572, 159, 38), (572, 160, 37), (573, 161, 36), (573, 162, 35), (573, 163, 35), (574, 164, 33), (575, 165, 32), (576, 166, 30), (578, 167, 27), (580, 168, 24), (583, 169, 20), (585, 170, 18), (588, 171, 14), (592, 172, 8)], ['599,172,592,172,585,170,582,168,576,166,570,156,566,145,566,137,563,127,560,121,551,112,547,109,539,106,530,98,527,97,511,80,510,80,501,71,500,71,491,62,490,59,491,52,503,39,508,37,518,29,523,27,533,26,534,25,541,24,558,24,559,25,571,25,572,26,594,26,595,25,613,25,614,24,632,24,634,29,634,51,635,52,635,111,634,112,634,129,633,133,628,139,623,141,618,148,609,158,606,165']), (957285035, 492601069, 445, 277, 480, 2, 56, 0.9309285, [(322, 2, 72), (290, 3, 137), (284, 4, 149), (280, 5, 155), (279, 6, 157), (279, 7, 159), (278, 8, 161), (278, 9, 163), (278, 10, 165), (278, 11, 167), (278, 12, 169), (278, 13, 171), (278, 14, 173), (278, 15, 174), (278, 16, 176), (278, 17, 177), (278, 18, 179), (278, 19, 180), (278, 20, 181), (278, 21, 182), (278, 22, 184), (278, 23, 185), (278, 24, 186), (278, 25, 187), (278, 26, 187), (278, 27, 188), (278, 28, 189), (278, 29, 189), (279, 30, 189), (280, 31, 32), (339, 31, 131), (369, 32, 101), (400, 33, 71), (410, 34, 61), (419, 35, 53), (424, 36, 48), (428, 37, 45), (431, 38, 42), (434, 39, 40), (436, 40, 39), (438, 41, 38), (442, 42, 35), (445, 43, 32), (448, 44, 29), (451, 45, 26), (454, 46, 23), (459, 47, 18), (463, 48, 13), (466, 49, 8)], ['473,49,466,49,462,47,454,46,450,44,445,43,441,41,438,41,433,38,431,38,427,36,424,36,423,35,419,35,418,34,410,34,409,33,400,33,399,32,369,32,368,31,339,31,338,30,312,30,311,31,280,31,278,29,278,8,280,5,283,5,284,4,289,4,290,3,321,3,322,2,393,2,394,3,426,3,427,4,432,4,439,9,450,14,452,16,456,18,464,25,466,29,469,31,472,38,476,42,476,47']), (957285035, 492601069, 445, 459, 552, 9, 49, 0.5945563, [(519, 10, 9), (484, 11, 47), (460, 12, 6), (479, 12, 64), (460, 13, 7), (473, 13, 71), (460, 14, 84), (460, 15, 84), (460, 16, 85), (460, 17, 85), (460, 18, 86), (461, 19, 85), (462, 20, 83), (463, 21, 78), (469, 22, 64), (469, 23, 58), (470, 24, 54), (471, 25, 52), (470, 26, 50), (471, 27, 48), (471, 28, 47), (471, 29, 45), (472, 30, 42), (473, 31, 40), (473, 32, 39), (474, 33, 37), (475, 34, 35), (476, 35, 32), (476, 36, 31), (476, 37, 30), (477, 38, 27), (477, 39, 25), (478, 40, 22), (478, 41, 20), (479, 42, 17), (480, 43, 9), (492, 43, 2), (483, 44, 1)], ['483,44,482,43,480,43,478,41,476,37,476,35,471,29,471,27,470,26,471,25,469,23,469,22,463,21,460,18,460,12,465,12,467,14,472,14,473,13,483,12,484,11,518,11,519,10,527,10,531,12,542,12,543,13,543,15,545,18,545,19,544,20,541,20,540,21,533,21,532,22,527,22,526,23,524,23,522,25,520,25,517,28,514,29,505,37,493,43,489,42,488,43,484,43'])], 'temp/1743654927_1794822_957285035_a42482e51c93c8025d243dd179aee85b.jpg']} free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4521 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.17461800575256348 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu Apr 3 06:36:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4331 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-04-03 06:36:04.401663: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-04-03 06:36:04.431130: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-04-03 06:36:04.433440: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb9bc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-04-03 06:36:04.433530: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-04-03 06:36:04.437735: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-04-03 06:36:04.568212: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2080b7e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-04-03 06:36:04.568330: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-04-03 06:36:04.569820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-03 06:36:04.570486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:36:04.575816: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:36:04.580465: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-03 06:36:04.581080: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-03 06:36:04.614957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-03 06:36:04.618569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-03 06:36:04.627371: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:36:04.629203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-03 06:36:04.629421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:36:04.630411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-04-03 06:36:04.630448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-04-03 06:36:04.630472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-04-03 06:36:04.632385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 356 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-04-03 06:36:04.766018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-03 06:36:04.766112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:36:04.766135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:36:04.766156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-03 06:36:04.766176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-03 06:36:04.766195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-03 06:36:04.766229: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-03 06:36:04.766250: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:36:04.767038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-03 06:36:04.767939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-03 06:36:04.767988: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:36:04.768017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:36:04.768040: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-03 06:36:04.768060: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-03 06:36:04.768078: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-03 06:36:04.768097: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-03 06:36:04.768117: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:36:04.768907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-03 06:36:04.768939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-04-03 06:36:04.768948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-04-03 06:36:04.768956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-04-03 06:36:04.769753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 356 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead 2025-04-03 06:36:17.479894: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 49.00MiB (rounded to 51380224) Current allocation summary follows. 2025-04-03 06:36:17.479977: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-04-03 06:36:17.479997: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 77, Chunks in use: 77. 19.2KiB allocated for chunks. 19.2KiB in use in bin. 9.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480014: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480030: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 259, Chunks in use: 259. 259.2KiB allocated for chunks. 259.2KiB in use in bin. 259.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480047: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 56, Chunks in use: 56. 112.0KiB allocated for chunks. 112.0KiB in use in bin. 112.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480063: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 120, Chunks in use: 120. 487.5KiB allocated for chunks. 487.5KiB in use in bin. 480.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480097: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 23, Chunks in use: 21. 194.8KiB allocated for chunks. 172.0KiB in use in bin. 172.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480114: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 2, Chunks in use: 2. 40.0KiB allocated for chunks. 40.0KiB in use in bin. 40.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480130: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 2, Chunks in use: 1. 84.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-04-03 06:36:17.480145: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480161: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 5, Chunks in use: 4. 783.2KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-04-03 06:36:17.480177: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 9, Chunks in use: 8. 2.66MiB allocated for chunks. 2.16MiB in use in bin. 2.00MiB client-requested in use in bin. 2025-04-03 06:36:17.480192: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 8, Chunks in use: 7. 4.64MiB allocated for chunks. 3.94MiB in use in bin. 3.75MiB client-requested in use in bin. 2025-04-03 06:36:17.480208: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 47, Chunks in use: 46. 53.12MiB allocated for chunks. 51.88MiB in use in bin. 46.00MiB client-requested in use in bin. 2025-04-03 06:36:17.480224: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 31, Chunks in use: 29. 71.00MiB allocated for chunks. 66.25MiB in use in bin. 64.50MiB client-requested in use in bin. 2025-04-03 06:36:17.480239: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 10, Chunks in use: 7. 46.25MiB allocated for chunks. 30.50MiB in use in bin. 26.75MiB client-requested in use in bin. 2025-04-03 06:36:17.480255: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 4, Chunks in use: 4. 35.00MiB allocated for chunks. 35.00MiB in use in bin. 35.00MiB client-requested in use in bin. 2025-04-03 06:36:17.480269: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-04-03 06:36:17.480285: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 3, Chunks in use: 2. 141.88MiB allocated for chunks. 101.88MiB in use in bin. 98.00MiB client-requested in use in bin. 2025-04-03 06:36:17.480299: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-04-03 06:36:17.480313: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-04-03 06:36:17.480327: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-04-03 06:36:17.480341: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 49.00MiB was 32.00MiB, Chunk State: 2025-04-03 06:36:17.480362: I tensorflow/core/common_runtime/bfc_allocator.cc:963] Size: 40.00MiB | Requested Size: 0B | in_use: 0 | bin_num: 17, prev: Size: 4.50MiB | Requested Size: 4.50MiB | in_use: 1 | bin_num: -1 2025-04-03 06:36:17.480384: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 106823680 2025-04-03 06:36:17.480400: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7c6000000 of size 51380224 next 699 2025-04-03 06:36:17.480413: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7c9100000 of size 55443456 next 18446744073709551615 2025-04-03 06:36:17.480425: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 134217728 2025-04-03 06:36:17.480438: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7ce000000 of size 4194304 next 632 2025-04-03 06:36:17.480450: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7ce400000 of size 4194304 next 617 2025-04-03 06:36:17.480462: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7ce800000 of size 4194304 next 616 2025-04-03 06:36:17.480475: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7cec00000 of size 6291456 next 608 2025-04-03 06:36:17.480488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7cf200000 of size 9437184 next 607 2025-04-03 06:36:17.480500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7cfb00000 of size 4194304 next 644 2025-04-03 06:36:17.480512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7cff00000 of size 4194304 next 625 2025-04-03 06:36:17.480524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d0300000 of size 8388608 next 624 2025-04-03 06:36:17.480536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d0b00000 of size 4194304 next 661 2025-04-03 06:36:17.480548: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d0f00000 of size 2359296 next 676 2025-04-03 06:36:17.480561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d1140000 of size 2359296 next 678 2025-04-03 06:36:17.480573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d1380000 of size 2359296 next 680 2025-04-03 06:36:17.480585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7d15c0000 of size 7602176 next 639 2025-04-03 06:36:17.480597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d1d00000 of size 9437184 next 638 2025-04-03 06:36:17.480609: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d2600000 of size 9437184 next 656 2025-04-03 06:36:17.480621: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7d2f00000 of size 4718592 next 685 2025-04-03 06:36:17.480633: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d3380000 of size 4718592 next 684 2025-04-03 06:36:17.480645: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7d3800000 of size 41943040 next 18446744073709551615 2025-04-03 06:36:17.480657: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 67108864 2025-04-03 06:36:17.480670: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d6000000 of size 2359296 next 370 2025-04-03 06:36:17.480682: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d6240000 of size 1048576 next 394 2025-04-03 06:36:17.480694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7d6340000 of size 1310720 next 388 2025-04-03 06:36:17.480706: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d6480000 of size 2359296 next 387 2025-04-03 06:36:17.480718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d66c0000 of size 1048576 next 412 2025-04-03 06:36:17.480731: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d67c0000 of size 1310720 next 406 2025-04-03 06:36:17.480743: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d6900000 of size 2359296 next 405 2025-04-03 06:36:17.480755: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d6b40000 of size 1048576 next 430 2025-04-03 06:36:17.480767: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d6c40000 of size 1310720 next 424 2025-04-03 06:36:17.480788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d6d80000 of size 2359296 next 423 2025-04-03 06:36:17.480801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d6fc0000 of size 1048576 next 447 2025-04-03 06:36:17.480813: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d70c0000 of size 1310720 next 441 2025-04-03 06:36:17.480825: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d7200000 of size 2359296 next 440 2025-04-03 06:36:17.480837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d7440000 of size 1048576 next 465 2025-04-03 06:36:17.480849: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d7540000 of size 1310720 next 459 2025-04-03 06:36:17.480861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d7680000 of size 2359296 next 458 2025-04-03 06:36:17.480873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d78c0000 of size 1048576 next 483 2025-04-03 06:36:17.480885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d79c0000 of size 1310720 next 477 2025-04-03 06:36:17.480897: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d7b00000 of size 2359296 next 476 2025-04-03 06:36:17.480909: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d7d40000 of size 1048576 next 501 2025-04-03 06:36:17.480921: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d7e40000 of size 1310720 next 495 2025-04-03 06:36:17.480934: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d7f80000 of size 2359296 next 494 2025-04-03 06:36:17.480946: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d81c0000 of size 1048576 next 519 2025-04-03 06:36:17.480958: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d82c0000 of size 1310720 next 513 2025-04-03 06:36:17.480970: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d8400000 of size 2359296 next 512 2025-04-03 06:36:17.480982: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d8640000 of size 1048576 next 537 2025-04-03 06:36:17.480994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d8740000 of size 1310720 next 531 2025-04-03 06:36:17.481006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d8880000 of size 2359296 next 530 2025-04-03 06:36:17.481018: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d8ac0000 of size 1048576 next 555 2025-04-03 06:36:17.481031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d8bc0000 of size 1310720 next 549 2025-04-03 06:36:17.481043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d8d00000 of size 2359296 next 548 2025-04-03 06:36:17.481055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d8f40000 of size 1048576 next 573 2025-04-03 06:36:17.481067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d9040000 of size 1310720 next 567 2025-04-03 06:36:17.481079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d9180000 of size 2359296 next 566 2025-04-03 06:36:17.481091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d93c0000 of size 1048576 next 591 2025-04-03 06:36:17.481103: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d94c0000 of size 1310720 next 585 2025-04-03 06:36:17.481115: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d9600000 of size 2359296 next 584 2025-04-03 06:36:17.481127: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7d9840000 of size 2097152 next 597 2025-04-03 06:36:17.481140: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d9a40000 of size 2097152 next 596 2025-04-03 06:36:17.481152: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7d9c40000 of size 3932160 next 18446744073709551615 2025-04-03 06:36:17.481172: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432 2025-04-03 06:36:17.481185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7da000000 of size 2359296 next 245 2025-04-03 06:36:17.481197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7da240000 of size 1048576 next 269 2025-04-03 06:36:17.481209: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7da340000 of size 1310720 next 263 2025-04-03 06:36:17.481221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7da480000 of size 2359296 next 262 2025-04-03 06:36:17.481234: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7da6c0000 of size 1048576 next 287 2025-04-03 06:36:17.481246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7da7c0000 of size 1310720 next 281 2025-04-03 06:36:17.481258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7da900000 of size 2359296 next 280 2025-04-03 06:36:17.481270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dab40000 of size 1048576 next 304 2025-04-03 06:36:17.481282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dac40000 of size 1310720 next 298 2025-04-03 06:36:17.481294: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dad80000 of size 2359296 next 297 2025-04-03 06:36:17.481306: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dafc0000 of size 1048576 next 322 2025-04-03 06:36:17.481318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7db0c0000 of size 1310720 next 316 2025-04-03 06:36:17.481330: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7db200000 of size 2359296 next 315 2025-04-03 06:36:17.481342: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7db440000 of size 1048576 next 340 2025-04-03 06:36:17.481354: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7db540000 of size 1310720 next 334 2025-04-03 06:36:17.481366: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7db680000 of size 2359296 next 333 2025-04-03 06:36:17.481378: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7db8c0000 of size 1048576 next 358 2025-04-03 06:36:17.481390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7db9c0000 of size 1310720 next 352 2025-04-03 06:36:17.481402: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dbb00000 of size 2359296 next 351 2025-04-03 06:36:17.481414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dbd40000 of size 1048576 next 376 2025-04-03 06:36:17.481427: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dbe40000 of size 1835008 next 18446744073709551615 2025-04-03 06:36:17.481439: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216 2025-04-03 06:36:17.481451: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dc000000 of size 2359296 next 181 2025-04-03 06:36:17.481463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dc240000 of size 2097152 next 196 2025-04-03 06:36:17.481475: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dc440000 of size 1048576 next 216 2025-04-03 06:36:17.481487: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dc540000 of size 1310720 next 209 2025-04-03 06:36:17.481499: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dc680000 of size 2359296 next 208 2025-04-03 06:36:17.481511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dc8c0000 of size 1048576 next 234 2025-04-03 06:36:17.481523: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dc9c0000 of size 1310720 next 228 2025-04-03 06:36:17.481535: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dcb00000 of size 2359296 next 227 2025-04-03 06:36:17.481547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7dcd40000 of size 2883584 next 18446744073709551615 2025-04-03 06:36:17.481568: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-04-03 06:36:17.481581: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de600000 of size 147456 next 55 2025-04-03 06:36:17.481593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de624000 of size 65536 next 78 2025-04-03 06:36:17.481606: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de634000 of size 4096 next 191 2025-04-03 06:36:17.481618: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de635000 of size 4096 next 192 2025-04-03 06:36:17.481630: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de636000 of size 4096 next 193 2025-04-03 06:36:17.481642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de637000 of size 256 next 194 2025-04-03 06:36:17.481655: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de637100 of size 256 next 195 2025-04-03 06:36:17.481667: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de637200 of size 4096 next 197 2025-04-03 06:36:17.481679: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de638200 of size 4096 next 198 2025-04-03 06:36:17.481691: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de639200 of size 4096 next 199 2025-04-03 06:36:17.481703: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63a200 of size 4096 next 200 2025-04-03 06:36:17.481715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63b200 of size 4096 next 201 2025-04-03 06:36:17.481728: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63c200 of size 1024 next 202 2025-04-03 06:36:17.481740: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63c600 of size 1024 next 204 2025-04-03 06:36:17.481752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63ca00 of size 1024 next 205 2025-04-03 06:36:17.481764: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63ce00 of size 1024 next 206 2025-04-03 06:36:17.481776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63d200 of size 1024 next 207 2025-04-03 06:36:17.481788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63d600 of size 1024 next 210 2025-04-03 06:36:17.481800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63da00 of size 1024 next 211 2025-04-03 06:36:17.481812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63de00 of size 1024 next 212 2025-04-03 06:36:17.481825: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63e200 of size 1024 next 213 2025-04-03 06:36:17.481837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63e600 of size 1024 next 214 2025-04-03 06:36:17.481849: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63ea00 of size 4096 next 215 2025-04-03 06:36:17.481861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de63fa00 of size 4096 next 217 2025-04-03 06:36:17.481873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de640a00 of size 4096 next 218 2025-04-03 06:36:17.481885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de641a00 of size 4096 next 219 2025-04-03 06:36:17.481897: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de642a00 of size 4096 next 220 2025-04-03 06:36:17.481909: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de643a00 of size 1024 next 221 2025-04-03 06:36:17.481922: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de643e00 of size 1024 next 223 2025-04-03 06:36:17.481934: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de644200 of size 1024 next 224 2025-04-03 06:36:17.481946: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de644600 of size 1024 next 225 2025-04-03 06:36:17.481967: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de644a00 of size 1024 next 226 2025-04-03 06:36:17.481979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de644e00 of size 1024 next 229 2025-04-03 06:36:17.481991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de645200 of size 1024 next 230 2025-04-03 06:36:17.482004: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de645600 of size 1024 next 231 2025-04-03 06:36:17.482016: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de645a00 of size 1024 next 232 2025-04-03 06:36:17.482028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de645e00 of size 1024 next 233 2025-04-03 06:36:17.482040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de646200 of size 7680 next 72 2025-04-03 06:36:17.482052: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de648000 of size 147456 next 71 2025-04-03 06:36:17.482065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de66c000 of size 131072 next 87 2025-04-03 06:36:17.482077: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de68c000 of size 4096 next 289 2025-04-03 06:36:17.482089: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de68d000 of size 4096 next 290 2025-04-03 06:36:17.482101: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de68e000 of size 4096 next 291 2025-04-03 06:36:17.482113: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de68f000 of size 1024 next 292 2025-04-03 06:36:17.482125: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de68f400 of size 1024 next 293 2025-04-03 06:36:17.482137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de68f800 of size 1024 next 294 2025-04-03 06:36:17.482149: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de68fc00 of size 1024 next 295 2025-04-03 06:36:17.482161: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de690000 of size 1024 next 296 2025-04-03 06:36:17.482173: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de690400 of size 1024 next 299 2025-04-03 06:36:17.482185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de690800 of size 1024 next 300 2025-04-03 06:36:17.482197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de690c00 of size 1024 next 301 2025-04-03 06:36:17.482209: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de691000 of size 1024 next 302 2025-04-03 06:36:17.482222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de691400 of size 1024 next 303 2025-04-03 06:36:17.482234: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de691800 of size 4096 next 305 2025-04-03 06:36:17.482246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de692800 of size 4096 next 306 2025-04-03 06:36:17.482258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de693800 of size 4096 next 307 2025-04-03 06:36:17.482270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de694800 of size 4096 next 308 2025-04-03 06:36:17.482282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de695800 of size 4096 next 309 2025-04-03 06:36:17.482294: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de696800 of size 1024 next 310 2025-04-03 06:36:17.482306: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de696c00 of size 1024 next 311 2025-04-03 06:36:17.482318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de697000 of size 1024 next 312 2025-04-03 06:36:17.482330: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de697400 of size 1024 next 313 2025-04-03 06:36:17.482342: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de697800 of size 1024 next 314 2025-04-03 06:36:17.482362: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de697c00 of size 1024 next 317 2025-04-03 06:36:17.482375: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de698000 of size 1024 next 318 2025-04-03 06:36:17.482387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de698400 of size 1024 next 319 2025-04-03 06:36:17.482399: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de698800 of size 1024 next 320 2025-04-03 06:36:17.482411: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de698c00 of size 1024 next 321 2025-04-03 06:36:17.482423: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de699000 of size 4096 next 323 2025-04-03 06:36:17.482435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69a000 of size 4096 next 324 2025-04-03 06:36:17.482447: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69b000 of size 4096 next 325 2025-04-03 06:36:17.482460: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69c000 of size 4096 next 326 2025-04-03 06:36:17.482472: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69d000 of size 4096 next 327 2025-04-03 06:36:17.482484: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69e000 of size 1024 next 328 2025-04-03 06:36:17.482496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69e400 of size 1024 next 329 2025-04-03 06:36:17.482508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69e800 of size 1024 next 330 2025-04-03 06:36:17.482520: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69ec00 of size 1024 next 331 2025-04-03 06:36:17.482532: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69f000 of size 1024 next 332 2025-04-03 06:36:17.482544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69f400 of size 1024 next 335 2025-04-03 06:36:17.482556: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69f800 of size 1024 next 336 2025-04-03 06:36:17.482568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de69fc00 of size 1024 next 337 2025-04-03 06:36:17.482580: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a0000 of size 1024 next 338 2025-04-03 06:36:17.482592: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a0400 of size 1024 next 339 2025-04-03 06:36:17.482604: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a0800 of size 4096 next 341 2025-04-03 06:36:17.482617: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a1800 of size 4096 next 342 2025-04-03 06:36:17.482629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a2800 of size 4096 next 343 2025-04-03 06:36:17.482641: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a3800 of size 4096 next 344 2025-04-03 06:36:17.482653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a4800 of size 4096 next 345 2025-04-03 06:36:17.482665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a5800 of size 1024 next 346 2025-04-03 06:36:17.482677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a5c00 of size 1024 next 347 2025-04-03 06:36:17.482689: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a6000 of size 1024 next 348 2025-04-03 06:36:17.482701: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a6400 of size 1024 next 349 2025-04-03 06:36:17.482713: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a6800 of size 1024 next 350 2025-04-03 06:36:17.482726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a6c00 of size 1024 next 353 2025-04-03 06:36:17.482738: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a7000 of size 1024 next 354 2025-04-03 06:36:17.482758: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a7400 of size 1024 next 355 2025-04-03 06:36:17.482770: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a7800 of size 1024 next 356 2025-04-03 06:36:17.482782: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a7c00 of size 1024 next 357 2025-04-03 06:36:17.482794: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a8000 of size 4096 next 359 2025-04-03 06:36:17.482807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6a9000 of size 4096 next 360 2025-04-03 06:36:17.482819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6aa000 of size 4096 next 361 2025-04-03 06:36:17.482831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ab000 of size 4096 next 362 2025-04-03 06:36:17.482843: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ac000 of size 4096 next 363 2025-04-03 06:36:17.482855: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ad000 of size 1024 next 364 2025-04-03 06:36:17.482867: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ad400 of size 1024 next 365 2025-04-03 06:36:17.482879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ad800 of size 1024 next 366 2025-04-03 06:36:17.482891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6adc00 of size 1024 next 367 2025-04-03 06:36:17.482918: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ae000 of size 1024 next 368 2025-04-03 06:36:17.482931: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ae400 of size 1024 next 371 2025-04-03 06:36:17.482943: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ae800 of size 1024 next 372 2025-04-03 06:36:17.482955: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6aec00 of size 1024 next 373 2025-04-03 06:36:17.482967: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6af000 of size 1024 next 374 2025-04-03 06:36:17.482979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6af400 of size 1024 next 375 2025-04-03 06:36:17.482991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6af800 of size 4096 next 377 2025-04-03 06:36:17.483003: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b0800 of size 4096 next 378 2025-04-03 06:36:17.483015: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b1800 of size 4096 next 379 2025-04-03 06:36:17.483027: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b2800 of size 4096 next 380 2025-04-03 06:36:17.483039: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b3800 of size 4096 next 381 2025-04-03 06:36:17.483051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b4800 of size 1024 next 382 2025-04-03 06:36:17.483063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b4c00 of size 1024 next 383 2025-04-03 06:36:17.483076: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b5000 of size 1024 next 384 2025-04-03 06:36:17.483088: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b5400 of size 1024 next 385 2025-04-03 06:36:17.483100: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b5800 of size 1024 next 386 2025-04-03 06:36:17.483112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b5c00 of size 1024 next 389 2025-04-03 06:36:17.483124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b6000 of size 1024 next 390 2025-04-03 06:36:17.483136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b6400 of size 1024 next 391 2025-04-03 06:36:17.483148: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b6800 of size 1024 next 392 2025-04-03 06:36:17.483160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b6c00 of size 1024 next 393 2025-04-03 06:36:17.483181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b7000 of size 4096 next 395 2025-04-03 06:36:17.483193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b8000 of size 4096 next 396 2025-04-03 06:36:17.483205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6b9000 of size 4096 next 397 2025-04-03 06:36:17.483218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ba000 of size 4096 next 398 2025-04-03 06:36:17.483230: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bb000 of size 4096 next 399 2025-04-03 06:36:17.483242: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bc000 of size 1024 next 400 2025-04-03 06:36:17.483254: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bc400 of size 1024 next 401 2025-04-03 06:36:17.483266: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bc800 of size 1024 next 402 2025-04-03 06:36:17.483278: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bcc00 of size 1024 next 403 2025-04-03 06:36:17.483290: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bd000 of size 1024 next 404 2025-04-03 06:36:17.483302: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bd400 of size 1024 next 407 2025-04-03 06:36:17.483314: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bd800 of size 1024 next 408 2025-04-03 06:36:17.483326: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bdc00 of size 1024 next 409 2025-04-03 06:36:17.483338: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6be000 of size 1024 next 410 2025-04-03 06:36:17.483350: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6be400 of size 1024 next 411 2025-04-03 06:36:17.483362: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6be800 of size 4096 next 413 2025-04-03 06:36:17.483374: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6bf800 of size 4096 next 414 2025-04-03 06:36:17.483386: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c0800 of size 4096 next 415 2025-04-03 06:36:17.483398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c1800 of size 4096 next 416 2025-04-03 06:36:17.483410: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c2800 of size 4096 next 417 2025-04-03 06:36:17.483423: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c3800 of size 1024 next 418 2025-04-03 06:36:17.483435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c3c00 of size 1024 next 419 2025-04-03 06:36:17.483447: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c4000 of size 1024 next 420 2025-04-03 06:36:17.483459: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c4400 of size 1024 next 421 2025-04-03 06:36:17.483471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c4800 of size 1024 next 422 2025-04-03 06:36:17.483483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c4c00 of size 1024 next 425 2025-04-03 06:36:17.483495: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c5000 of size 1024 next 426 2025-04-03 06:36:17.483507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c5400 of size 1024 next 427 2025-04-03 06:36:17.483519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c5800 of size 1024 next 428 2025-04-03 06:36:17.483531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c5c00 of size 1024 next 429 2025-04-03 06:36:17.483543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c6000 of size 4096 next 431 2025-04-03 06:36:17.483555: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c7000 of size 4096 next 432 2025-04-03 06:36:17.483575: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c8000 of size 4096 next 433 2025-04-03 06:36:17.483588: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6c9000 of size 4096 next 434 2025-04-03 06:36:17.483600: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6ca000 of size 4096 next 435 2025-04-03 06:36:17.483612: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6cb000 of size 1024 next 436 2025-04-03 06:36:17.483624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6cb400 of size 1024 next 437 2025-04-03 06:36:17.483636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6cb800 of size 1024 next 438 2025-04-03 06:36:17.483648: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6cbc00 of size 1024 next 103 2025-04-03 06:36:17.483661: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de6cc000 of size 262144 next 102 2025-04-03 06:36:17.483673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de70c000 of size 262144 next 674 2025-04-03 06:36:17.483686: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7de74c000 of size 737280 next 18446744073709551615 2025-04-03 06:36:17.483698: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-04-03 06:36:17.483710: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de800000 of size 589824 next 96 2025-04-03 06:36:17.483722: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de890000 of size 262144 next 117 2025-04-03 06:36:17.483734: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d0000 of size 1024 next 439 2025-04-03 06:36:17.483747: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d0400 of size 1024 next 442 2025-04-03 06:36:17.483759: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d0800 of size 1024 next 443 2025-04-03 06:36:17.483771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d0c00 of size 1024 next 444 2025-04-03 06:36:17.483783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d1000 of size 1024 next 445 2025-04-03 06:36:17.483795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d1400 of size 1024 next 446 2025-04-03 06:36:17.483807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d1800 of size 4096 next 448 2025-04-03 06:36:17.483819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d2800 of size 4096 next 449 2025-04-03 06:36:17.483831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d3800 of size 4096 next 450 2025-04-03 06:36:17.483843: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d4800 of size 4096 next 451 2025-04-03 06:36:17.483855: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d5800 of size 4096 next 452 2025-04-03 06:36:17.483868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d6800 of size 1024 next 453 2025-04-03 06:36:17.483880: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d6c00 of size 1024 next 454 2025-04-03 06:36:17.483892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d7000 of size 1024 next 455 2025-04-03 06:36:17.483904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d7400 of size 1024 next 456 2025-04-03 06:36:17.483916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d7800 of size 1024 next 457 2025-04-03 06:36:17.483928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d7c00 of size 1024 next 460 2025-04-03 06:36:17.483940: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d8000 of size 1024 next 461 2025-04-03 06:36:17.483952: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d8400 of size 1024 next 462 2025-04-03 06:36:17.483972: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d8800 of size 1024 next 463 2025-04-03 06:36:17.483985: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d8c00 of size 1024 next 464 2025-04-03 06:36:17.483997: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8d9000 of size 4096 next 466 2025-04-03 06:36:17.484009: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8da000 of size 4096 next 467 2025-04-03 06:36:17.484021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8db000 of size 4096 next 468 2025-04-03 06:36:17.484033: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8dc000 of size 4096 next 469 2025-04-03 06:36:17.484045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8dd000 of size 4096 next 470 2025-04-03 06:36:17.484057: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8de000 of size 1024 next 471 2025-04-03 06:36:17.484070: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8de400 of size 1024 next 472 2025-04-03 06:36:17.484082: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8de800 of size 1024 next 473 2025-04-03 06:36:17.484094: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8dec00 of size 1024 next 474 2025-04-03 06:36:17.484106: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8df000 of size 1024 next 475 2025-04-03 06:36:17.484118: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8df400 of size 1024 next 478 2025-04-03 06:36:17.484130: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8df800 of size 1024 next 479 2025-04-03 06:36:17.484142: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8dfc00 of size 1024 next 480 2025-04-03 06:36:17.484154: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e0000 of size 1024 next 481 2025-04-03 06:36:17.484166: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e0400 of size 1024 next 482 2025-04-03 06:36:17.484178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e0800 of size 4096 next 484 2025-04-03 06:36:17.484190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e1800 of size 4096 next 485 2025-04-03 06:36:17.484202: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e2800 of size 4096 next 486 2025-04-03 06:36:17.484214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e3800 of size 4096 next 487 2025-04-03 06:36:17.484226: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e4800 of size 4096 next 488 2025-04-03 06:36:17.484238: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e5800 of size 1024 next 489 2025-04-03 06:36:17.484250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e5c00 of size 1024 next 490 2025-04-03 06:36:17.484263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e6000 of size 1024 next 491 2025-04-03 06:36:17.484275: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e6400 of size 1024 next 492 2025-04-03 06:36:17.484287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e6800 of size 1024 next 493 2025-04-03 06:36:17.484299: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e6c00 of size 1024 next 496 2025-04-03 06:36:17.484311: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e7000 of size 1024 next 497 2025-04-03 06:36:17.484323: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e7400 of size 1024 next 498 2025-04-03 06:36:17.484335: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e7800 of size 1024 next 499 2025-04-03 06:36:17.484347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e7c00 of size 1024 next 500 2025-04-03 06:36:17.484367: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e8000 of size 4096 next 502 2025-04-03 06:36:17.484380: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8e9000 of size 4096 next 503 2025-04-03 06:36:17.484392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ea000 of size 4096 next 504 2025-04-03 06:36:17.484404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8eb000 of size 4096 next 505 2025-04-03 06:36:17.484416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ec000 of size 4096 next 506 2025-04-03 06:36:17.484428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ed000 of size 1024 next 507 2025-04-03 06:36:17.484440: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ed400 of size 1024 next 508 2025-04-03 06:36:17.484452: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ed800 of size 1024 next 509 2025-04-03 06:36:17.484464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8edc00 of size 1024 next 510 2025-04-03 06:36:17.484476: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ee000 of size 1024 next 511 2025-04-03 06:36:17.484488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ee400 of size 1024 next 514 2025-04-03 06:36:17.484500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ee800 of size 1024 next 515 2025-04-03 06:36:17.484512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8eec00 of size 1024 next 516 2025-04-03 06:36:17.484524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ef000 of size 1024 next 517 2025-04-03 06:36:17.484536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ef400 of size 1024 next 518 2025-04-03 06:36:17.484548: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ef800 of size 4096 next 520 2025-04-03 06:36:17.484561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f0800 of size 4096 next 521 2025-04-03 06:36:17.484573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f1800 of size 4096 next 522 2025-04-03 06:36:17.484585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f2800 of size 4096 next 523 2025-04-03 06:36:17.484597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f3800 of size 4096 next 524 2025-04-03 06:36:17.484609: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f4800 of size 1024 next 525 2025-04-03 06:36:17.484621: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f4c00 of size 1024 next 526 2025-04-03 06:36:17.484633: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f5000 of size 1024 next 527 2025-04-03 06:36:17.484645: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f5400 of size 1024 next 528 2025-04-03 06:36:17.484657: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f5800 of size 1024 next 529 2025-04-03 06:36:17.484669: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f5c00 of size 1024 next 532 2025-04-03 06:36:17.484681: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f6000 of size 1024 next 533 2025-04-03 06:36:17.484693: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f6400 of size 1024 next 534 2025-04-03 06:36:17.484705: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f6800 of size 1024 next 535 2025-04-03 06:36:17.484717: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f6c00 of size 1024 next 536 2025-04-03 06:36:17.484729: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f7000 of size 4096 next 538 2025-04-03 06:36:17.484741: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f8000 of size 4096 next 539 2025-04-03 06:36:17.484754: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8f9000 of size 4096 next 540 2025-04-03 06:36:17.484774: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fa000 of size 4096 next 541 2025-04-03 06:36:17.484786: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fb000 of size 4096 next 542 2025-04-03 06:36:17.484798: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fc000 of size 1024 next 543 2025-04-03 06:36:17.484811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fc400 of size 1024 next 544 2025-04-03 06:36:17.484823: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fc800 of size 1024 next 545 2025-04-03 06:36:17.484835: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fcc00 of size 1024 next 546 2025-04-03 06:36:17.484847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fd000 of size 1024 next 547 2025-04-03 06:36:17.484859: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fd400 of size 1024 next 550 2025-04-03 06:36:17.484871: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fd800 of size 1024 next 551 2025-04-03 06:36:17.484883: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fdc00 of size 1024 next 552 2025-04-03 06:36:17.484899: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fe000 of size 1024 next 553 2025-04-03 06:36:17.484915: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fe400 of size 1024 next 554 2025-04-03 06:36:17.484933: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8fe800 of size 4096 next 556 2025-04-03 06:36:17.484950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de8ff800 of size 4096 next 557 2025-04-03 06:36:17.484968: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de900800 of size 4096 next 558 2025-04-03 06:36:17.484985: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de901800 of size 4096 next 559 2025-04-03 06:36:17.484999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de902800 of size 4096 next 560 2025-04-03 06:36:17.485012: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de903800 of size 1024 next 561 2025-04-03 06:36:17.485024: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de903c00 of size 1024 next 562 2025-04-03 06:36:17.485036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de904000 of size 1024 next 563 2025-04-03 06:36:17.485048: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de904400 of size 1024 next 564 2025-04-03 06:36:17.485060: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de904800 of size 1024 next 565 2025-04-03 06:36:17.485072: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de904c00 of size 1024 next 568 2025-04-03 06:36:17.485084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de905000 of size 1024 next 569 2025-04-03 06:36:17.485096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de905400 of size 1024 next 570 2025-04-03 06:36:17.485108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de905800 of size 1024 next 571 2025-04-03 06:36:17.485120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de905c00 of size 1024 next 572 2025-04-03 06:36:17.485132: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de906000 of size 4096 next 574 2025-04-03 06:36:17.485144: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de907000 of size 4096 next 575 2025-04-03 06:36:17.485156: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de908000 of size 4096 next 576 2025-04-03 06:36:17.485168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de909000 of size 4096 next 577 2025-04-03 06:36:17.485180: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90a000 of size 4096 next 578 2025-04-03 06:36:17.485203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90b000 of size 1024 next 579 2025-04-03 06:36:17.485215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90b400 of size 1024 next 580 2025-04-03 06:36:17.485227: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90b800 of size 1024 next 581 2025-04-03 06:36:17.485239: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90bc00 of size 1024 next 582 2025-04-03 06:36:17.485251: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90c000 of size 1024 next 583 2025-04-03 06:36:17.485263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90c400 of size 1024 next 586 2025-04-03 06:36:17.485276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90c800 of size 1024 next 587 2025-04-03 06:36:17.485288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90cc00 of size 1024 next 588 2025-04-03 06:36:17.485300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90d000 of size 1024 next 589 2025-04-03 06:36:17.485312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90d400 of size 1024 next 590 2025-04-03 06:36:17.485324: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90d800 of size 4096 next 592 2025-04-03 06:36:17.485336: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de90e800 of size 6144 next 112 2025-04-03 06:36:17.485349: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de910000 of size 524288 next 111 2025-04-03 06:36:17.485361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de990000 of size 262144 next 135 2025-04-03 06:36:17.485373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7de9d0000 of size 327680 next 124 2025-04-03 06:36:17.485386: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dea20000 of size 589824 next 123 2025-04-03 06:36:17.485398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7deab0000 of size 262144 next 152 2025-04-03 06:36:17.485410: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7deaf0000 of size 327680 next 142 2025-04-03 06:36:17.485422: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7deb40000 of size 786432 next 18446744073709551615 2025-04-03 06:36:17.485434: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-04-03 06:36:17.485446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec00000 of size 4096 next 593 2025-04-03 06:36:17.485459: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec01000 of size 4096 next 594 2025-04-03 06:36:17.485471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec02000 of size 4096 next 595 2025-04-03 06:36:17.485483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec03000 of size 2048 next 598 2025-04-03 06:36:17.485495: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec03800 of size 2048 next 599 2025-04-03 06:36:17.485507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec04000 of size 2048 next 600 2025-04-03 06:36:17.485519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec04800 of size 2048 next 601 2025-04-03 06:36:17.485531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec05000 of size 2048 next 602 2025-04-03 06:36:17.485543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec05800 of size 256 next 605 2025-04-03 06:36:17.485555: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec05900 of size 256 next 606 2025-04-03 06:36:17.485567: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec05a00 of size 2048 next 604 2025-04-03 06:36:17.485579: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec06200 of size 2048 next 609 2025-04-03 06:36:17.485603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec06a00 of size 2048 next 610 2025-04-03 06:36:17.485616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec07200 of size 2048 next 611 2025-04-03 06:36:17.485628: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec07a00 of size 2048 next 612 2025-04-03 06:36:17.485640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec08200 of size 256 next 614 2025-04-03 06:36:17.485652: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec08300 of size 256 next 615 2025-04-03 06:36:17.485664: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec08400 of size 8192 next 613 2025-04-03 06:36:17.485676: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec0a400 of size 8192 next 618 2025-04-03 06:36:17.485688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec0c400 of size 8192 next 619 2025-04-03 06:36:17.485700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec0e400 of size 8192 next 620 2025-04-03 06:36:17.485712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec10400 of size 8192 next 621 2025-04-03 06:36:17.485724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec12400 of size 256 next 622 2025-04-03 06:36:17.485736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec12500 of size 256 next 623 2025-04-03 06:36:17.485748: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec12600 of size 8192 next 626 2025-04-03 06:36:17.485760: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec14600 of size 8192 next 627 2025-04-03 06:36:17.485772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec16600 of size 8192 next 628 2025-04-03 06:36:17.485784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec18600 of size 8192 next 629 2025-04-03 06:36:17.485796: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1a600 of size 8192 next 630 2025-04-03 06:36:17.485808: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1c600 of size 2048 next 631 2025-04-03 06:36:17.485820: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1ce00 of size 2048 next 633 2025-04-03 06:36:17.485832: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1d600 of size 2048 next 634 2025-04-03 06:36:17.485844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1de00 of size 2048 next 635 2025-04-03 06:36:17.485856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1e600 of size 2048 next 636 2025-04-03 06:36:17.485868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1ee00 of size 2048 next 637 2025-04-03 06:36:17.485880: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1f600 of size 2048 next 640 2025-04-03 06:36:17.485892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec1fe00 of size 2048 next 641 2025-04-03 06:36:17.485904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec20600 of size 2048 next 642 2025-04-03 06:36:17.485916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec20e00 of size 2048 next 643 2025-04-03 06:36:17.485928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec21600 of size 8192 next 645 2025-04-03 06:36:17.485940: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec23600 of size 8192 next 646 2025-04-03 06:36:17.485952: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec25600 of size 8192 next 647 2025-04-03 06:36:17.485964: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec27600 of size 8192 next 648 2025-04-03 06:36:17.485976: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec29600 of size 8192 next 649 2025-04-03 06:36:17.485996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2b600 of size 2048 next 650 2025-04-03 06:36:17.486008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2be00 of size 2048 next 651 2025-04-03 06:36:17.486020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2c600 of size 2048 next 652 2025-04-03 06:36:17.486032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2ce00 of size 2048 next 653 2025-04-03 06:36:17.486044: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2d600 of size 2048 next 654 2025-04-03 06:36:17.486056: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2de00 of size 2048 next 655 2025-04-03 06:36:17.486068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2e600 of size 2048 next 657 2025-04-03 06:36:17.486080: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2ee00 of size 2048 next 658 2025-04-03 06:36:17.486092: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2f600 of size 2048 next 659 2025-04-03 06:36:17.486104: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec2fe00 of size 2048 next 660 2025-04-03 06:36:17.486116: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec30600 of size 8192 next 662 2025-04-03 06:36:17.486128: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec32600 of size 8192 next 663 2025-04-03 06:36:17.486140: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec34600 of size 8192 next 664 2025-04-03 06:36:17.486152: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec36600 of size 8192 next 665 2025-04-03 06:36:17.486164: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec38600 of size 8192 next 666 2025-04-03 06:36:17.486176: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3a600 of size 1024 next 667 2025-04-03 06:36:17.486188: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3aa00 of size 1024 next 668 2025-04-03 06:36:17.486200: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3ae00 of size 1024 next 669 2025-04-03 06:36:17.486212: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3b200 of size 1024 next 673 2025-04-03 06:36:17.486224: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3b600 of size 1024 next 675 2025-04-03 06:36:17.486236: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3ba00 of size 1024 next 677 2025-04-03 06:36:17.486248: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3be00 of size 1024 next 679 2025-04-03 06:36:17.486260: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3c200 of size 1024 next 681 2025-04-03 06:36:17.486272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3c600 of size 256 next 682 2025-04-03 06:36:17.486284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3c700 of size 256 next 683 2025-04-03 06:36:17.486296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3c800 of size 2048 next 686 2025-04-03 06:36:17.486308: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3d000 of size 256 next 690 2025-04-03 06:36:17.486320: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3d100 of size 256 next 694 2025-04-03 06:36:17.486331: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3d200 of size 256 next 695 2025-04-03 06:36:17.486343: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3d300 of size 256 next 693 2025-04-03 06:36:17.486355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3d400 of size 256 next 700 2025-04-03 06:36:17.486367: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec3d500 of size 256 next 701 2025-04-03 06:36:17.486379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7dec3d600 of size 10752 next 687 2025-04-03 06:36:17.486399: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec40000 of size 256 next 688 2025-04-03 06:36:17.486412: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec40100 of size 256 next 689 2025-04-03 06:36:17.486424: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7dec40200 of size 12544 next 691 2025-04-03 06:36:17.486436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec43300 of size 12288 next 692 2025-04-03 06:36:17.486448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7dec46300 of size 49152 next 697 2025-04-03 06:36:17.486461: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec52300 of size 24576 next 696 2025-04-03 06:36:17.486473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7dec58300 of size 228608 next 160 2025-04-03 06:36:17.486485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dec90000 of size 589824 next 158 2025-04-03 06:36:17.486497: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7ded20000 of size 524288 next 171 2025-04-03 06:36:17.486509: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7deda0000 of size 256 next 671 2025-04-03 06:36:17.486521: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7deda0100 of size 256 next 672 2025-04-03 06:36:17.486533: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fb7deda0200 of size 523776 next 670 2025-04-03 06:36:17.486545: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7dee20000 of size 524288 next 222 2025-04-03 06:36:17.486557: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7deea0000 of size 1048576 next 190 2025-04-03 06:36:17.486569: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7defa0000 of size 1048576 next 189 2025-04-03 06:36:17.486581: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7df0a0000 of size 1048576 next 203 2025-04-03 06:36:17.486593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7df1a0000 of size 1048576 next 251 2025-04-03 06:36:17.486605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb7df2a0000 of size 1441792 next 18446744073709551615 2025-04-03 06:36:17.486617: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-04-03 06:36:17.486630: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00000 of size 1280 next 1 2025-04-03 06:36:17.486642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00500 of size 256 next 5 2025-04-03 06:36:17.486654: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00600 of size 256 next 8 2025-04-03 06:36:17.486666: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00700 of size 256 next 9 2025-04-03 06:36:17.486678: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00800 of size 256 next 10 2025-04-03 06:36:17.486690: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00900 of size 256 next 11 2025-04-03 06:36:17.486702: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00a00 of size 256 next 12 2025-04-03 06:36:17.486714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00b00 of size 256 next 13 2025-04-03 06:36:17.486726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00c00 of size 256 next 17 2025-04-03 06:36:17.486738: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00d00 of size 256 next 19 2025-04-03 06:36:17.486750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00e00 of size 256 next 20 2025-04-03 06:36:17.486762: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e00f00 of size 256 next 21 2025-04-03 06:36:17.486774: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01000 of size 256 next 22 2025-04-03 06:36:17.486794: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01100 of size 256 next 24 2025-04-03 06:36:17.486807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01200 of size 256 next 25 2025-04-03 06:36:17.486819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01300 of size 256 next 23 2025-04-03 06:36:17.486831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01400 of size 256 next 28 2025-04-03 06:36:17.486843: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01500 of size 256 next 29 2025-04-03 06:36:17.486855: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01600 of size 256 next 30 2025-04-03 06:36:17.486867: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01700 of size 256 next 31 2025-04-03 06:36:17.486879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01800 of size 256 next 33 2025-04-03 06:36:17.486891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01900 of size 256 next 34 2025-04-03 06:36:17.486913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01a00 of size 1024 next 32 2025-04-03 06:36:17.486927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e01e00 of size 1024 next 37 2025-04-03 06:36:17.486941: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e02200 of size 1024 next 38 2025-04-03 06:36:17.486953: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e02600 of size 1024 next 39 2025-04-03 06:36:17.486965: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e02a00 of size 1024 next 40 2025-04-03 06:36:17.486977: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e02e00 of size 1024 next 41 2025-04-03 06:36:17.486989: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e03200 of size 1024 next 43 2025-04-03 06:36:17.487001: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e03600 of size 1024 next 44 2025-04-03 06:36:17.487013: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e03a00 of size 1024 next 45 2025-04-03 06:36:17.487025: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e03e00 of size 1024 next 46 2025-04-03 06:36:17.487037: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04200 of size 256 next 48 2025-04-03 06:36:17.487049: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04300 of size 256 next 49 2025-04-03 06:36:17.487061: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04400 of size 256 next 50 2025-04-03 06:36:17.487073: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04500 of size 256 next 51 2025-04-03 06:36:17.487085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04600 of size 256 next 52 2025-04-03 06:36:17.487097: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04700 of size 256 next 53 2025-04-03 06:36:17.487109: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04800 of size 256 next 56 2025-04-03 06:36:17.487121: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04900 of size 256 next 57 2025-04-03 06:36:17.487133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04a00 of size 256 next 58 2025-04-03 06:36:17.487145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04b00 of size 256 next 14 2025-04-03 06:36:17.487157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04c00 of size 256 next 15 2025-04-03 06:36:17.487168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04d00 of size 256 next 16 2025-04-03 06:36:17.487180: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e04e00 of size 1024 next 60 2025-04-03 06:36:17.487202: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e05200 of size 1024 next 61 2025-04-03 06:36:17.487214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e05600 of size 1024 next 62 2025-04-03 06:36:17.487226: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e05a00 of size 1024 next 63 2025-04-03 06:36:17.487238: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e05e00 of size 1024 next 64 2025-04-03 06:36:17.487250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06200 of size 256 next 66 2025-04-03 06:36:17.487262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06300 of size 256 next 67 2025-04-03 06:36:17.487274: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06400 of size 256 next 68 2025-04-03 06:36:17.487286: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06500 of size 256 next 69 2025-04-03 06:36:17.487298: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06600 of size 256 next 70 2025-04-03 06:36:17.487310: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06700 of size 256 next 73 2025-04-03 06:36:17.487322: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06800 of size 256 next 74 2025-04-03 06:36:17.487334: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06900 of size 256 next 75 2025-04-03 06:36:17.487346: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06a00 of size 256 next 76 2025-04-03 06:36:17.487358: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06b00 of size 256 next 77 2025-04-03 06:36:17.487370: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e06c00 of size 1024 next 79 2025-04-03 06:36:17.487382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e07000 of size 1024 next 80 2025-04-03 06:36:17.487394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e07400 of size 1024 next 81 2025-04-03 06:36:17.487406: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e07800 of size 1024 next 82 2025-04-03 06:36:17.487418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e07c00 of size 1024 next 83 2025-04-03 06:36:17.487430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08000 of size 256 next 85 2025-04-03 06:36:17.487442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08100 of size 256 next 86 2025-04-03 06:36:17.487455: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08200 of size 512 next 84 2025-04-03 06:36:17.487467: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08400 of size 512 next 88 2025-04-03 06:36:17.487479: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08600 of size 512 next 89 2025-04-03 06:36:17.487491: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08800 of size 512 next 90 2025-04-03 06:36:17.487502: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08a00 of size 512 next 91 2025-04-03 06:36:17.487514: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08c00 of size 256 next 93 2025-04-03 06:36:17.487526: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08d00 of size 256 next 94 2025-04-03 06:36:17.487539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e08e00 of size 512 next 92 2025-04-03 06:36:17.487551: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e09000 of size 512 next 97 2025-04-03 06:36:17.487562: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e09200 of size 512 next 98 2025-04-03 06:36:17.487574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e09400 of size 512 next 99 2025-04-03 06:36:17.487586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e09600 of size 512 next 2 2025-04-03 06:36:17.487606: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e09800 of size 256 next 3 2025-04-03 06:36:17.487619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e09900 of size 256 next 4 2025-04-03 06:36:17.487632: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e09a00 of size 16384 next 18 2025-04-03 06:36:17.487644: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e0da00 of size 256 next 100 2025-04-03 06:36:17.487656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e0db00 of size 256 next 101 2025-04-03 06:36:17.487668: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e0dc00 of size 2048 next 104 2025-04-03 06:36:17.487680: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e0e400 of size 2048 next 105 2025-04-03 06:36:17.487692: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e0ec00 of size 2048 next 106 2025-04-03 06:36:17.487704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e0f400 of size 2048 next 107 2025-04-03 06:36:17.487716: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e0fc00 of size 2048 next 108 2025-04-03 06:36:17.487728: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e10400 of size 256 next 109 2025-04-03 06:36:17.487740: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e10500 of size 256 next 110 2025-04-03 06:36:17.487752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e10600 of size 2048 next 113 2025-04-03 06:36:17.487764: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e10e00 of size 2048 next 114 2025-04-03 06:36:17.487776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e11600 of size 2048 next 115 2025-04-03 06:36:17.487788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e11e00 of size 2048 next 116 2025-04-03 06:36:17.487800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e12600 of size 2048 next 6 2025-04-03 06:36:17.487812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e12e00 of size 37632 next 7 2025-04-03 06:36:17.487824: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1c100 of size 512 next 118 2025-04-03 06:36:17.487836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1c300 of size 512 next 119 2025-04-03 06:36:17.487848: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1c500 of size 512 next 120 2025-04-03 06:36:17.487860: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1c700 of size 512 next 121 2025-04-03 06:36:17.487872: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1c900 of size 512 next 122 2025-04-03 06:36:17.487884: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1cb00 of size 512 next 125 2025-04-03 06:36:17.487896: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1cd00 of size 512 next 126 2025-04-03 06:36:17.487908: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1cf00 of size 512 next 127 2025-04-03 06:36:17.487920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1d100 of size 512 next 128 2025-04-03 06:36:17.487932: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1d300 of size 512 next 129 2025-04-03 06:36:17.487944: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1d500 of size 2048 next 130 2025-04-03 06:36:17.487956: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1dd00 of size 2048 next 131 2025-04-03 06:36:17.487968: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1e500 of size 2048 next 132 2025-04-03 06:36:17.487980: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1ed00 of size 2048 next 133 2025-04-03 06:36:17.487992: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1f500 of size 2048 next 134 2025-04-03 06:36:17.488012: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1fd00 of size 512 next 136 2025-04-03 06:36:17.488024: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e1ff00 of size 512 next 137 2025-04-03 06:36:17.488036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e20100 of size 512 next 138 2025-04-03 06:36:17.488048: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e20300 of size 512 next 139 2025-04-03 06:36:17.488060: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e20500 of size 512 next 140 2025-04-03 06:36:17.488072: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e20700 of size 512 next 141 2025-04-03 06:36:17.488084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e20900 of size 512 next 143 2025-04-03 06:36:17.488096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e20b00 of size 512 next 144 2025-04-03 06:36:17.488108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e20d00 of size 512 next 145 2025-04-03 06:36:17.488120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e20f00 of size 512 next 146 2025-04-03 06:36:17.488132: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e21100 of size 2048 next 147 2025-04-03 06:36:17.488145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e21900 of size 2048 next 148 2025-04-03 06:36:17.488159: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e22100 of size 2048 next 149 2025-04-03 06:36:17.488171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e22900 of size 2048 next 150 2025-04-03 06:36:17.488183: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e23100 of size 2048 next 151 2025-04-03 06:36:17.488197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e23900 of size 512 next 153 2025-04-03 06:36:17.488210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e23b00 of size 512 next 154 2025-04-03 06:36:17.488222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e23d00 of size 512 next 155 2025-04-03 06:36:17.488234: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e23f00 of size 512 next 156 2025-04-03 06:36:17.488245: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e24100 of size 512 next 157 2025-04-03 06:36:17.488257: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e24300 of size 512 next 161 2025-04-03 06:36:17.488269: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e24500 of size 512 next 162 2025-04-03 06:36:17.488281: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e24700 of size 512 next 163 2025-04-03 06:36:17.488293: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e24900 of size 512 next 164 2025-04-03 06:36:17.488305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e24b00 of size 512 next 165 2025-04-03 06:36:17.488317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e24d00 of size 2048 next 166 2025-04-03 06:36:17.488329: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e25500 of size 2048 next 167 2025-04-03 06:36:17.488341: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e25d00 of size 2048 next 168 2025-04-03 06:36:17.488353: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e26500 of size 2048 next 169 2025-04-03 06:36:17.488366: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e26d00 of size 2048 next 170 2025-04-03 06:36:17.488378: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e27500 of size 1024 next 172 2025-04-03 06:36:17.488390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e27900 of size 1024 next 173 2025-04-03 06:36:17.488410: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e27d00 of size 1024 next 174 2025-04-03 06:36:17.488423: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e28100 of size 1024 next 175 2025-04-03 06:36:17.488435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e28500 of size 1024 next 176 2025-04-03 06:36:17.488447: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e28900 of size 256 next 178 2025-04-03 06:36:17.488459: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e28a00 of size 256 next 179 2025-04-03 06:36:17.488471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e28b00 of size 1024 next 177 2025-04-03 06:36:17.488483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e28f00 of size 1024 next 182 2025-04-03 06:36:17.488495: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e29300 of size 1024 next 183 2025-04-03 06:36:17.488507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e29700 of size 1024 next 184 2025-04-03 06:36:17.488519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e29b00 of size 1024 next 185 2025-04-03 06:36:17.488531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e29f00 of size 256 next 187 2025-04-03 06:36:17.488543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e2a000 of size 256 next 188 2025-04-03 06:36:17.488555: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e2a100 of size 4096 next 186 2025-04-03 06:36:17.488567: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e2b100 of size 4096 next 42 2025-04-03 06:36:17.488579: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e2c100 of size 65536 next 36 2025-04-03 06:36:17.488591: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e3c100 of size 65536 next 35 2025-04-03 06:36:17.488603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e4c100 of size 4096 next 235 2025-04-03 06:36:17.488615: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e4d100 of size 4096 next 236 2025-04-03 06:36:17.488627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e4e100 of size 4096 next 237 2025-04-03 06:36:17.488639: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e4f100 of size 4096 next 238 2025-04-03 06:36:17.488651: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e50100 of size 1024 next 239 2025-04-03 06:36:17.488663: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e50500 of size 1024 next 240 2025-04-03 06:36:17.488675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e50900 of size 1024 next 241 2025-04-03 06:36:17.488687: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e50d00 of size 1024 next 242 2025-04-03 06:36:17.488699: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e51100 of size 1024 next 243 2025-04-03 06:36:17.488711: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e51500 of size 1024 next 246 2025-04-03 06:36:17.488723: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e51900 of size 1024 next 247 2025-04-03 06:36:17.488735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e51d00 of size 1024 next 248 2025-04-03 06:36:17.488747: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e52100 of size 1024 next 249 2025-04-03 06:36:17.488759: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e52500 of size 1024 next 250 2025-04-03 06:36:17.488771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e52900 of size 4096 next 252 2025-04-03 06:36:17.488783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e53900 of size 4096 next 253 2025-04-03 06:36:17.488803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e54900 of size 4096 next 254 2025-04-03 06:36:17.488816: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e55900 of size 4096 next 255 2025-04-03 06:36:17.488827: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e56900 of size 4096 next 256 2025-04-03 06:36:17.488839: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e57900 of size 1024 next 257 2025-04-03 06:36:17.488851: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e57d00 of size 1024 next 258 2025-04-03 06:36:17.488863: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e58100 of size 1024 next 259 2025-04-03 06:36:17.488875: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e58500 of size 1024 next 260 2025-04-03 06:36:17.488887: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e58900 of size 1024 next 261 2025-04-03 06:36:17.488899: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e58d00 of size 1024 next 264 2025-04-03 06:36:17.488911: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e59100 of size 1024 next 265 2025-04-03 06:36:17.488923: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e59500 of size 1024 next 266 2025-04-03 06:36:17.488935: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e59900 of size 1024 next 267 2025-04-03 06:36:17.488947: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e59d00 of size 1024 next 268 2025-04-03 06:36:17.488959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5a100 of size 4096 next 270 2025-04-03 06:36:17.488971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5b100 of size 4096 next 271 2025-04-03 06:36:17.488983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5c100 of size 4096 next 272 2025-04-03 06:36:17.488995: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5d100 of size 4096 next 273 2025-04-03 06:36:17.489007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5e100 of size 4096 next 274 2025-04-03 06:36:17.489019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5f100 of size 1024 next 275 2025-04-03 06:36:17.489031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5f500 of size 1024 next 276 2025-04-03 06:36:17.489043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5f900 of size 1024 next 277 2025-04-03 06:36:17.489055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e5fd00 of size 1024 next 278 2025-04-03 06:36:17.489067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e60100 of size 1024 next 279 2025-04-03 06:36:17.489079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e60500 of size 1024 next 282 2025-04-03 06:36:17.489091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e60900 of size 1024 next 283 2025-04-03 06:36:17.489103: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e60d00 of size 1024 next 284 2025-04-03 06:36:17.489114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e61100 of size 1024 next 285 2025-04-03 06:36:17.489126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e61500 of size 1024 next 286 2025-04-03 06:36:17.489138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e61900 of size 4096 next 288 2025-04-03 06:36:17.489150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e62900 of size 6144 next 27 2025-04-03 06:36:17.489162: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e64100 of size 147456 next 26 2025-04-03 06:36:17.489174: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e88100 of size 65536 next 47 2025-04-03 06:36:17.489186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916e98100 of size 65536 next 59 2025-04-03 06:36:17.489206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916ea8100 of size 65536 next 65 2025-04-03 06:36:17.489220: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb916eb8100 of size 294656 next 18446744073709551615 2025-04-03 06:36:17.489231: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-04-03 06:36:17.489248: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 77 Chunks of size 256 totalling 19.2KiB 2025-04-03 06:36:17.489262: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-04-03 06:36:17.489275: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 258 Chunks of size 1024 totalling 258.0KiB 2025-04-03 06:36:17.489289: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-04-03 06:36:17.489302: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 56 Chunks of size 2048 totalling 112.0KiB 2025-04-03 06:36:17.489315: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 117 Chunks of size 4096 totalling 468.0KiB 2025-04-03 06:36:17.489329: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 6144 totalling 12.0KiB 2025-04-03 06:36:17.489342: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2025-04-03 06:36:17.489355: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 20 Chunks of size 8192 totalling 160.0KiB 2025-04-03 06:36:17.489369: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 12288 totalling 12.0KiB 2025-04-03 06:36:17.489382: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-04-03 06:36:17.489396: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 24576 totalling 24.0KiB 2025-04-03 06:36:17.489409: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-04-03 06:36:17.489422: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-04-03 06:36:17.489435: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-04-03 06:36:17.489449: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-04-03 06:36:17.489462: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 5 Chunks of size 262144 totalling 1.25MiB 2025-04-03 06:36:17.489475: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-04-03 06:36:17.489489: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-04-03 06:36:17.489502: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 524288 totalling 1.50MiB 2025-04-03 06:36:17.489515: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-04-03 06:36:17.489528: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-04-03 06:36:17.489542: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1048576 totalling 25.00MiB 2025-04-03 06:36:17.489555: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 19 Chunks of size 1310720 totalling 23.75MiB 2025-04-03 06:36:17.489569: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB 2025-04-03 06:36:17.489582: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB 2025-04-03 06:36:17.489595: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 2097152 totalling 4.00MiB 2025-04-03 06:36:17.489608: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 26 Chunks of size 2359296 totalling 58.50MiB 2025-04-03 06:36:17.489621: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 3932160 totalling 3.75MiB 2025-04-03 06:36:17.489644: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 5 Chunks of size 4194304 totalling 20.00MiB 2025-04-03 06:36:17.489657: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 4718592 totalling 4.50MiB 2025-04-03 06:36:17.489671: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 6291456 totalling 6.00MiB 2025-04-03 06:36:17.489684: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 8388608 totalling 8.00MiB 2025-04-03 06:36:17.489697: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 9437184 totalling 27.00MiB 2025-04-03 06:36:17.489710: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 51380224 totalling 49.00MiB 2025-04-03 06:36:17.489724: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 55443456 totalling 52.88MiB 2025-04-03 06:36:17.489737: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 293.63MiB 2025-04-03 06:36:17.489750: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 374210560 memory_limit_: 374210560 available bytes: 0 curr_region_allocation_bytes_: 536870912 2025-04-03 06:36:17.489766: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 374210560 InUse: 307898880 MaxInUse: 307899136 NumAllocs: 2327 MaxAllocSize: 55443456 2025-04-03 06:36:17.489798: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ****************************x***************_*********__________************************************ 2025-04-03 06:36:17.489843: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[7,7,256,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[7,7,256,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] we want to redo the detection max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 ERROR : 'int' object is not subscriptable reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: "'int' object is not subscriptable" /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_47a3e4a81bc81b59dfbf2b2b102e93cf6aa9db97 SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_47a3e4a81bc81b59dfbf2b2b102e93cf6aa9db97','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/April/03042025/python_test3//data_2/data_log/job/2025/April/03042025/python_test3/log-python3----short_python3--v--marlene-06:35:00.txt','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.28511643409729004 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Thu Apr 3 06:36:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1743654977_1794822_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1743654977_1794822_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 (640, 960, 3) time for calcul the mask position with numpy : 0.002745389938354492 nb_pixel_total : 5616 time to create 1 rle with old method : 0.018334627151489258 time for calcul the mask position with numpy : 0.0017504692077636719 nb_pixel_total : 11852 time to create 1 rle with old method : 0.025799036026000977 time for calcul the mask position with numpy : 0.001989603042602539 nb_pixel_total : 13914 time to create 1 rle with old method : 0.025793790817260742 time for calcul the mask position with numpy : 0.0018460750579833984 nb_pixel_total : 6643 time to create 1 rle with old method : 0.012788057327270508 time for calcul the mask position with numpy : 0.002025604248046875 nb_pixel_total : 3758 time to create 1 rle with old method : 0.014507293701171875 time for calcul the mask position with numpy : 0.004003763198852539 nb_pixel_total : 83831 time to create 1 rle with old method : 0.15128016471862793 time for calcul the mask position with numpy : 0.0024993419647216797 nb_pixel_total : 38207 time to create 1 rle with old method : 0.059450387954711914 time for calcul the mask position with numpy : 0.0018160343170166016 nb_pixel_total : 29399 time to create 1 rle with old method : 0.03647899627685547 time for calcul the mask position with numpy : 0.0015828609466552734 nb_pixel_total : 5256 time to create 1 rle with old method : 0.006497383117675781 time for calcul the mask position with numpy : 0.001596689224243164 nb_pixel_total : 2328 time to create 1 rle with old method : 0.0027892589569091797 time for calcul the mask position with numpy : 0.0016632080078125 nb_pixel_total : 10805 time to create 1 rle with old method : 0.012392759323120117 time for calcul the mask position with numpy : 0.0016369819641113281 nb_pixel_total : 2937 time to create 1 rle with old method : 0.0035610198974609375 time for calcul the mask position with numpy : 0.001689910888671875 nb_pixel_total : 16868 time to create 1 rle with old method : 0.0200960636138916 time for calcul the mask position with numpy : 0.001628875732421875 nb_pixel_total : 4272 time to create 1 rle with old method : 0.005088329315185547 time for calcul the mask position with numpy : 0.001753091812133789 nb_pixel_total : 13098 time to create 1 rle with old method : 0.018120765686035156 time for calcul the mask position with numpy : 0.0016744136810302734 nb_pixel_total : 1215 time to create 1 rle with old method : 0.0015485286712646484 time for calcul the mask position with numpy : 0.0015406608581542969 nb_pixel_total : 2396 time to create 1 rle with old method : 0.0029664039611816406 time for calcul the mask position with numpy : 0.0015828609466552734 nb_pixel_total : 3951 time to create 1 rle with old method : 0.0047032833099365234 time for calcul the mask position with numpy : 0.0016279220581054688 nb_pixel_total : 16364 time to create 1 rle with old method : 0.020991802215576172 time for calcul the mask position with numpy : 0.0017817020416259766 nb_pixel_total : 3173 time to create 1 rle with old method : 0.003980875015258789 time for calcul the mask position with numpy : 0.0016970634460449219 nb_pixel_total : 2078 time to create 1 rle with old method : 0.004734516143798828 time for calcul the mask position with numpy : 0.0022644996643066406 nb_pixel_total : 7636 time to create 1 rle with old method : 0.009305477142333984 time for calcul the mask position with numpy : 0.0016925334930419922 nb_pixel_total : 886 time to create 1 rle with old method : 0.0011358261108398438 time for calcul the mask position with numpy : 0.0016186237335205078 nb_pixel_total : 3908 time to create 1 rle with old method : 0.00482487678527832 time for calcul the mask position with numpy : 0.0016324520111083984 nb_pixel_total : 4286 time to create 1 rle with old method : 0.005321502685546875 time for calcul the mask position with numpy : 0.001676797866821289 nb_pixel_total : 5680 time to create 1 rle with old method : 0.007325410842895508 time for calcul the mask position with numpy : 0.00193023681640625 nb_pixel_total : 3553 time to create 1 rle with old method : 0.00520014762878418 time for calcul the mask position with numpy : 0.001977682113647461 nb_pixel_total : 8639 time to create 1 rle with old method : 0.011469125747680664 time for calcul the mask position with numpy : 0.0016639232635498047 nb_pixel_total : 3304 time to create 1 rle with old method : 0.004385948181152344 time for calcul the mask position with numpy : 0.00156402587890625 nb_pixel_total : 2771 time to create 1 rle with old method : 0.0033447742462158203 time for calcul the mask position with numpy : 0.0016810894012451172 nb_pixel_total : 16456 time to create 1 rle with old method : 0.018725156784057617 time for calcul the mask position with numpy : 0.001638650894165039 nb_pixel_total : 11148 time to create 1 rle with old method : 0.01316976547241211 time for calcul the mask position with numpy : 0.0016474723815917969 nb_pixel_total : 2730 time to create 1 rle with old method : 0.0033538341522216797 time for calcul the mask position with numpy : 0.0015721321105957031 nb_pixel_total : 13067 time to create 1 rle with old method : 0.01609206199645996 time for calcul the mask position with numpy : 0.001542806625366211 nb_pixel_total : 1025 time to create 1 rle with old method : 0.0013442039489746094 time for calcul the mask position with numpy : 0.0015788078308105469 nb_pixel_total : 885 time to create 1 rle with old method : 0.0016334056854248047 time for calcul the mask position with numpy : 0.002362489700317383 nb_pixel_total : 2446 time to create 1 rle with old method : 0.0054378509521484375 time for calcul the mask position with numpy : 0.0019083023071289062 nb_pixel_total : 9869 time to create 1 rle with old method : 0.016108274459838867 time for calcul the mask position with numpy : 0.0018191337585449219 nb_pixel_total : 968 time to create 1 rle with old method : 0.0015833377838134766 time for calcul the mask position with numpy : 0.0017502307891845703 nb_pixel_total : 2407 time to create 1 rle with old method : 0.0040776729583740234 time for calcul the mask position with numpy : 0.0019104480743408203 nb_pixel_total : 18487 time to create 1 rle with old method : 0.027711153030395508 time for calcul the mask position with numpy : 0.001968860626220703 nb_pixel_total : 10622 time to create 1 rle with old method : 0.017457962036132812 time for calcul the mask position with numpy : 0.002105712890625 nb_pixel_total : 4126 time to create 1 rle with old method : 0.006940364837646484 time for calcul the mask position with numpy : 0.0019288063049316406 nb_pixel_total : 344 time to create 1 rle with old method : 0.0006353855133056641 time for calcul the mask position with numpy : 0.001924753189086914 nb_pixel_total : 1245 time to create 1 rle with old method : 0.002076864242553711 time for calcul the mask position with numpy : 0.0017549991607666016 nb_pixel_total : 1653 time to create 1 rle with old method : 0.00262451171875 time for calcul the mask position with numpy : 0.0017342567443847656 nb_pixel_total : 812 time to create 1 rle with old method : 0.0014181137084960938 time for calcul the mask position with numpy : 0.0017266273498535156 nb_pixel_total : 4191 time to create 1 rle with old method : 0.0066187381744384766 time for calcul the mask position with numpy : 0.0017883777618408203 nb_pixel_total : 858 time to create 1 rle with old method : 0.0015442371368408203 time for calcul the mask position with numpy : 0.0017502307891845703 nb_pixel_total : 586 time to create 1 rle with old method : 0.0009891986846923828 time for calcul the mask position with numpy : 0.0017838478088378906 nb_pixel_total : 1674 time to create 1 rle with old method : 0.0027549266815185547 time for calcul the mask position with numpy : 0.0018470287322998047 nb_pixel_total : 2388 time to create 1 rle with old method : 0.00410771369934082 time for calcul the mask position with numpy : 0.001861572265625 nb_pixel_total : 873 time to create 1 rle with old method : 0.0017173290252685547 time for calcul the mask position with numpy : 0.002071380615234375 nb_pixel_total : 14317 time to create 1 rle with old method : 0.022616147994995117 time for calcul the mask position with numpy : 0.0019068717956542969 nb_pixel_total : 8618 time to create 1 rle with old method : 0.013161182403564453 time for calcul the mask position with numpy : 0.0018169879913330078 nb_pixel_total : 2027 time to create 1 rle with old method : 0.0034236907958984375 time for calcul the mask position with numpy : 0.0019156932830810547 nb_pixel_total : 692 time to create 1 rle with old method : 0.0012359619140625 time for calcul the mask position with numpy : 0.002019166946411133 nb_pixel_total : 577 time to create 1 rle with old method : 0.0011165142059326172 time for calcul the mask position with numpy : 0.0019152164459228516 nb_pixel_total : 334 time to create 1 rle with old method : 0.0006494522094726562 time for calcul the mask position with numpy : 0.0019464492797851562 nb_pixel_total : 1709 time to create 1 rle with old method : 0.0029103755950927734 time for calcul the mask position with numpy : 0.002234935760498047 nb_pixel_total : 27683 time to create 1 rle with old method : 0.039063453674316406 time for calcul the mask position with numpy : 0.0015110969543457031 nb_pixel_total : 1206 time to create 1 rle with old method : 0.0015230178833007812 time for calcul the mask position with numpy : 0.001497507095336914 nb_pixel_total : 2770 time to create 1 rle with old method : 0.0041844844818115234 time for calcul the mask position with numpy : 0.002031087875366211 nb_pixel_total : 13024 time to create 1 rle with old method : 0.024306774139404297 time for calcul the mask position with numpy : 0.0018985271453857422 nb_pixel_total : 1075 time to create 1 rle with old method : 0.0022139549255371094 time for calcul the mask position with numpy : 0.0018551349639892578 nb_pixel_total : 1051 time to create 1 rle with old method : 0.0022385120391845703 time for calcul the mask position with numpy : 0.0018727779388427734 nb_pixel_total : 585 time to create 1 rle with old method : 0.0012485980987548828 time for calcul the mask position with numpy : 0.0018622875213623047 nb_pixel_total : 3093 time to create 1 rle with old method : 0.0061953067779541016 time for calcul the mask position with numpy : 0.001962423324584961 nb_pixel_total : 1795 time to create 1 rle with old method : 0.0034699440002441406 time for calcul the mask position with numpy : 0.001992940902709961 nb_pixel_total : 8439 time to create 1 rle with old method : 0.015897035598754883 time for calcul the mask position with numpy : 0.0024187564849853516 nb_pixel_total : 1515 time to create 1 rle with old method : 0.0031321048736572266 time for calcul the mask position with numpy : 0.0022318363189697266 nb_pixel_total : 616 time to create 1 rle with old method : 0.0012264251708984375 time for calcul the mask position with numpy : 0.0023872852325439453 nb_pixel_total : 16685 time to create 1 rle with old method : 0.032335519790649414 time for calcul the mask position with numpy : 0.0021486282348632812 nb_pixel_total : 9075 time to create 1 rle with old method : 0.011940479278564453 time for calcul the mask position with numpy : 0.0015227794647216797 nb_pixel_total : 713 time to create 1 rle with old method : 0.0010199546813964844 time for calcul the mask position with numpy : 0.0015573501586914062 nb_pixel_total : 1334 time to create 1 rle with old method : 0.0017914772033691406 time for calcul the mask position with numpy : 0.0015666484832763672 nb_pixel_total : 267 time to create 1 rle with old method : 0.00036907196044921875 time for calcul the mask position with numpy : 0.0015270709991455078 nb_pixel_total : 835 time to create 1 rle with old method : 0.0010609626770019531 time for calcul the mask position with numpy : 0.0015718936920166016 nb_pixel_total : 248 time to create 1 rle with old method : 0.0003726482391357422 time for calcul the mask position with numpy : 0.0016741752624511719 nb_pixel_total : 9492 time to create 1 rle with old method : 0.011352777481079102 time for calcul the mask position with numpy : 0.0017893314361572266 nb_pixel_total : 973 time to create 1 rle with old method : 0.0013942718505859375 time for calcul the mask position with numpy : 0.00156402587890625 nb_pixel_total : 221 time to create 1 rle with old method : 0.0003674030303955078 time for calcul the mask position with numpy : 0.0016598701477050781 nb_pixel_total : 595 time to create 1 rle with old method : 0.0008397102355957031 time for calcul the mask position with numpy : 0.0016222000122070312 nb_pixel_total : 1643 time to create 1 rle with old method : 0.0021834373474121094 time for calcul the mask position with numpy : 0.0016715526580810547 nb_pixel_total : 7497 time to create 1 rle with old method : 0.008859872817993164 time for calcul the mask position with numpy : 0.0015749931335449219 nb_pixel_total : 943 time to create 1 rle with old method : 0.0016376972198486328 time for calcul the mask position with numpy : 0.0015609264373779297 nb_pixel_total : 735 time to create 1 rle with old method : 0.001058340072631836 time for calcul the mask position with numpy : 0.0016252994537353516 nb_pixel_total : 292 time to create 1 rle with old method : 0.00047779083251953125 time for calcul the mask position with numpy : 0.0016350746154785156 nb_pixel_total : 7404 time to create 1 rle with old method : 0.009294271469116211 time for calcul the mask position with numpy : 0.001674652099609375 nb_pixel_total : 1484 time to create 1 rle with old method : 0.0018427371978759766 time for calcul the mask position with numpy : 0.0015797615051269531 nb_pixel_total : 5042 time to create 1 rle with old method : 0.00616765022277832 time for calcul the mask position with numpy : 0.0015904903411865234 nb_pixel_total : 1444 time to create 1 rle with old method : 0.0018451213836669922 time for calcul the mask position with numpy : 0.0015323162078857422 nb_pixel_total : 1124 time to create 1 rle with old method : 0.0014498233795166016 time for calcul the mask position with numpy : 0.0014948844909667969 nb_pixel_total : 484 time to create 1 rle with old method : 0.0006527900695800781 time for calcul the mask position with numpy : 0.0014984607696533203 nb_pixel_total : 890 time to create 1 rle with old method : 0.0011820793151855469 time for calcul the mask position with numpy : 0.0015964508056640625 nb_pixel_total : 2199 time to create 1 rle with old method : 0.003056764602661133 time for calcul the mask position with numpy : 0.0015017986297607422 nb_pixel_total : 1320 time to create 1 rle with old method : 0.0017235279083251953 time for calcul the mask position with numpy : 0.0014603137969970703 nb_pixel_total : 1426 time to create 1 rle with old method : 0.0018982887268066406 time for calcul the mask position with numpy : 0.0014564990997314453 nb_pixel_total : 1611 time to create 1 rle with old method : 0.0020160675048828125 time for calcul the mask position with numpy : 0.0014612674713134766 nb_pixel_total : 954 time to create 1 rle with old method : 0.00124359130859375 time for calcul the mask position with numpy : 0.001566171646118164 nb_pixel_total : 873 time to create 1 rle with old method : 0.0011098384857177734 insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) batch 1 Loaded 101 chid ids of type : 4677 Number RLEs to save : 9701 INSERT IGNORE INTO MTRPhoto.crop_segments (`crop_hashtag_id`, `x0`, `y0`, `length`) VALUES (%s, %s, %s , %s) first line : ('3745734087', '464', '201', '4') ... last line : ('3745734187', '815', '44', '6') INSERT IGNORE INTO MTRPhoto.crop_sum_segments (`crop_hashtag_id`, `sum_segments`) VALUES (%s, %s) TO DO : save crop sub photo not yet done ! After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : sam we use saveGeneral [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] Looping around the photos to save general results len do output : 1 /1189321094Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4573', None, None, None, None, None, None, None, None) ('4573', None, '1189321094', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', 'None', None, None, None, None, None)] time used for this insertion : 0.014484167098999023 save_final save missing photos in datou_result : time spend for datou_step_exec : 12.601986169815063 time spend to save output : 0.015001773834228516 total time spend for step 1 : 12.616987943649292 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'1189321094': [[, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], 'temp/1743654977_1794822_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png']} nb_objects detect : 101 ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.10310840606689453 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Thu Apr 3 06:36:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1743654990_1794822_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1743654990_1794822_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] local folder : /data/models_weight/detection_plaque_valcor_010622 /data/models_weight/detection_plaque_valcor_010622/caffemodel size_local : 349723073 size in s3 : 349723073 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:56 caffemodel already exist and didn't need to update /data/models_weight/detection_plaque_valcor_010622/test.prototxt size_local : 7163 size in s3 : 7163 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:55 test.prototxt already exist and didn't need to update prototxt : /data/models_weight/detection_plaque_valcor_010622/test.prototxt caffemodel : /data/models_weight/detection_plaque_valcor_010622/caffemodel Loaded network /data/models_weight/detection_plaque_valcor_010622/caffemodel About to compute detect_faster_rcnn : len(args) : 1 Inside frcnn step exec : nb paths : 1 image_path : temp/1743654990_1794822_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg image_size (600, 800, 3) [[[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] ... [[ 14 16 16] [ 13 15 15] [ 11 13 13] ... [198 206 205] [198 206 205] [198 206 205]] [[ 16 18 18] [ 14 16 16] [ 11 13 13] ... [206 214 213] [206 214 213] [206 214 213]] [[ 13 15 15] [ 12 14 14] [ 9 11 11] ... [210 218 217] [210 218 217] [210 218 217]]] Detection took 0.079s for 300 object proposals c : plaque list_crops.shape (72, 5) proba : 0.06385769 (374.13394, 293.91855, 430.819, 317.8074) proba : 0.05223692 (382.15323, 297.18018, 552.35144, 344.65063) proba : 0.012271373 (345.35672, 272.42072, 468.8627, 320.71777) We are managing local photo_id len de result frcnn : 1 After datou_step_exec type output : time spend for datou_step_exec : 2.633002281188965 time spend to save output : 7.295608520507812e-05 total time spend for step 1 : 2.63307523727417 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True Inside saveFrcnn : final : True verbose : True threshold to save the result : 0.1 output flattener : [(0, 493029425, 4370, 374, 430, 293, 317, 0.06385769, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.05223692, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012271373, None)] Warning : no hashtag_ids to insert in the database final : True begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', '0', 0, '0', 493061979, '0', None)] time used for this insertion : 0.012912988662719727 [917754606] map_info['map_portfolio_photo'] : {} final : True mtd_id 4184 list_pids : [917754606] Looping around the photos to save general results len do output : 1 /0 before output type Managing all output in save final without adding information in the mtr_datou_result ('4184', None, None, None, None, None, None, None, None) ('4184', None, '917754606', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', None, None, None, None, None, None)] time used for this insertion : 0.012109994888305664 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {0: [[(0, 493029425, 4370, 374, 430, 293, 317, 0.06385769, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.05223692, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012271373, None)], 'temp/1743654990_1794822_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg']} ############################### TEST thcl ################################ TEST THCL Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=2 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=2 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 2 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=2 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 1 thcl is not linked in the step_by_step architecture ! WARNING : step 2 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (916235064) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 916235064 download finish for photo 916235064 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.1283740997314453 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:thcl Thu Apr 3 06:36:33 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1743654993_1794822_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1743654993_1794822_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'355': 1} we are using the classfication for only one thcl 355 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (66, 66, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.008004903793334961 time to convert the images to numpy array : 0.003806591033935547 total time to convert the images to numpy array : 0.01228475570678711 list photo_ids error: [] list photo_ids correct : [916235064] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 355 To do loadFromThcl(), then load ParamDescType : thcl355 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (355) thcls : [{'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3390 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) To loadFromThcl() : net_3390 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1185 wait 20 seconds l 3637 free memory gpu now : 1185 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) param : , param.caffemodel : car_360_1027 None mean_file_type : mean_file_path : prototxt_file_path : model : car_360_1027 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : car_360_1027 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/car_360_1027 /data/models_weight/car_360_1027/caffemodel size_local : 542944640 size in s3 : 542944640 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 caffemodel already exist and didn't need to update /data/models_weight/car_360_1027/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy_fc.prototxt size_local : 1132 size in s3 : 1132 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy.prototxt size_local : 5654 size in s3 : 5654 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/mean.npy size_local : 1572944 size in s3 : 1572944 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:55 mean.npy already exist and didn't need to update /data/models_weight/car_360_1027/synset_words.txt size_local : 13687 size in s3 : 13687 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/car_360_1027/deploy.prototxt caffemodel_filename : /data/models_weight/car_360_1027/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 966 wait 20 seconds l 3637 free memory gpu now : 966 max_wait_temp : 1 max_wait : 0 dict_keys(['prob', 'pool5']) time used to do the prepocess of the images : 0.015942811965942383 time used to do the prediction : 0.16713976860046387 save descriptor for thcl : 355 (1, 512, 7, 7) Got the blobs of the net to insert : [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] code_as_byte_string:b'0000000000'| time to traite the descriptors : 0.06405162811279297 Testing : ['916235064'] In select_photos_meta_from_ids: SELECT photo_id, url, FROM_UNIXTIME(uploaded_at), latitude, longitude, text FROM MTRBack.photos WHERE photo_id IN (916235064) result : {916235064: {'photo_id': 916235064, 'url': 'https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/2017/10/14/6293d1bb790dc6902450e7c572b7d10b.jpg', 'latitude': 0.0, 'longitude': 0.0, 'text': None}} list_photo_exists : [916235064] storage_type for insertDescriptorsMulti : 1 To insert : 916235064 time to insert the descriptors : 1.5278918743133545 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : False verbose : True time used to find the portfolios of the photos select button_legend_list from MTRDatou.classification_theme where id = 355 SAVE THCL, output : {'916235064': [[('916235064', 'c_elysee_1027_gao__port_506302', 0.0018817182, 332, '355'), ('916235064', 'mokka_1027_gao__port_506374', 0.001163608, 332, '355'), ('916235064', 'captur_1027_gao__port_506399', 0.00081581634, 332, '355'), ('916235064', 'sorento_1027_gao__port_506192', 0.0011773179, 332, '355'), ('916235064', 'navara_1027_gao__port_506205', 0.002584883, 332, '355'), ('916235064', 'xc90_1027_gao__port_506350', 0.004170308, 332, '355'), ('916235064', 'saxo_1027_gao__port_506052', 0.0034805145, 332, '355'), ('916235064', 'trafic_1027_gao__port_506295', 0.007367851, 332, '355'), ('916235064', 'punto_evo_1027_gao__port_506066', 0.0021888907, 332, '355'), ('916235064', '5_1027_gao__port_506117', 0.0005798501, 332, '355'), ('916235064', '250_1027_gao__port_506065', 0.004590334, 332, '355'), ('916235064', 'd_max_1027_gao__port_506125', 0.0031583577, 332, '355'), ('916235064', 'panamera_1027_gao__port_506387', 0.0022508076, 332, '355'), ('916235064', 'alhambra_1027_gao__port_506381', 0.0053207735, 332, '355'), ('916235064', 'x6_1027_gao__port_506349', 0.0010999046, 332, '355'), ('916235064', 'vitara_1027_gao__port_506328', 0.005401945, 332, '355'), ('916235064', 'fiesta_1027_gao__port_506377', 0.003918739, 332, '355'), ('916235064', 'qashqai_1027_gao__port_506286', 0.0014788839, 332, '355'), ('916235064', '147_1027_gao__port_506124', 0.0019777853, 332, '355'), ('916235064', 'c5_1027_gao__port_506172', 0.0012442112, 332, '355'), ('916235064', 'q5_1027_gao__port_506206', 0.001505158, 332, '355'), ('916235064', 'giulia_1027_gao__port_506178', 0.0021694175, 332, '355'), ('916235064', 'karl_1027_gao__port_506371', 0.0027083026, 332, '355'), ('916235064', 'mehari_1027_gao__port_506076', 0.0047034635, 332, '355'), ('916235064', '911_1027_gao__port_506114', 0.0019418851, 332, '355'), ('916235064', '508_1027_gao__port_506329', 0.00095856783, 332, '355'), ('916235064', 'idea_1027_gao__port_506122', 0.0007700227, 332, '355'), ('916235064', 'megane_1027_gao__port_506220', 0.0019468981, 332, '355'), ('916235064', 'ghibli_1027_gao__port_506174', 0.0013726635, 332, '355'), ('916235064', 'touareg_1027_gao__port_506224', 0.0016202859, 332, '355'), ('916235064', 'i10_1027_gao__port_506232', 0.0013925632, 332, '355'), ('916235064', 'jumper_1027_gao__port_506234', 0.0100447675, 332, '355'), ('916235064', 'classe_clk_1027_gao__port_506173', 0.0010793622, 332, '355'), ('916235064', 'kuga_1027_gao__port_506181', 0.00084477605, 332, '355'), ('916235064', 'ct_1027_gao__port_506323', 0.0012520544, 332, '355'), ('916235064', 'leon_1027_gao__port_506326', 0.0025847263, 332, '355'), ('916235064', 'ds5_1027_gao__port_506376', 0.0012430203, 332, '355'), ('916235064', 'cordoba_1027_gao__port_506048', 0.002864923, 332, '355'), ('916235064', 'classe_cla_1027_gao__port_506400', 0.0012949633, 332, '355'), ('916235064', 'jumpy_1027_gao__port_506179', 0.010339891, 332, '355'), ('916235064', 'avensis_1027_gao__port_506311', 0.0018767754, 332, '355'), ('916235064', 'juke_1027_gao__port_506325', 0.0011344728, 332, '355'), ('916235064', '4008_1027_gao__port_506402', 0.0015758886, 332, '355'), ('916235064', '190_series_1027_gao__port_506051', 0.003979841, 332, '355'), ('916235064', 'serie_3_1027_gao__port_506294', 0.0028741036, 332, '355'), ('916235064', 'q7_1027_gao__port_506318', 0.002335566, 332, '355'), ('916235064', 'glc_1027_gao__port_506303', 0.0012106454, 332, '355'), ('916235064', 'grand_vitara_1027_gao__port_506175', 0.0011446713, 332, '355'), ('916235064', 's40_1027_gao__port_506099', 0.002233906, 332, '355'), ('916235064', 'toledo_1027_gao__port_506061', 0.0017464453, 332, '355'), ('916235064', '5008_1027_gao__port_506337', 0.0046997373, 332, '355'), ('916235064', 'continental_1027_gao__port_506250', 0.0021915927, 332, '355'), ('916235064', 'coupe_1027_gao__port_506082', 0.0022630256, 332, '355'), ('916235064', 'iq_1027_gao__port_506166', 0.0018174385, 332, '355'), ('916235064', '407_1027_gao__port_506133', 0.0009056405, 332, '355'), ('916235064', 'touran_1027_gao__port_506308', 0.0020405778, 332, '355'), ('916235064', '300c_1027_gao__port_506078', 0.0025333886, 332, '355'), ('916235064', 'classe_gl_1027_gao__port_506340', 0.004488921, 332, '355'), ('916235064', 'vivaro_1027_gao__port_506310', 0.0034256403, 332, '355'), ('916235064', 'sl_1027_gao__port_506100', 0.003135124, 332, '355'), ('916235064', 'elise_1027_gao__port_506121', 0.0010255396, 332, '355'), ('916235064', '1007_1027_gao__port_506070', 0.0015355027, 332, '355'), ('916235064', 'i40_1027_gao__port_506218', 0.000591493, 332, '355'), ('916235064', 'bipper_tepee_1027_gao__port_506227', 0.0040295757, 332, '355'), ('916235064', 'focus_1027_gao__port_506272', 0.0011586886, 332, '355'), ('916235064', 'primera_1027_gao__port_506147', 0.0012158157, 332, '355'), ('916235064', 'r4_1027_gao__port_506160', 0.014965846, 332, '355'), ('916235064', 'a8_1027_gao__port_506265', 0.0011320349, 332, '355'), ('916235064', 'boxer_1027_gao__port_506202', 0.0105460975, 332, '355'), ('916235064', 's5_1027_gao__port_506222', 0.0011985475, 332, '355'), ('916235064', 'r21_1027_gao__port_506093', 0.0041851415, 332, '355'), ('916235064', 'c3_1027_gao__port_506257', 0.0023634757, 332, '355'), ('916235064', 'santa_fe_1027_gao__port_506208', 0.0016324747, 332, '355'), ('916235064', 'm4_1027_gao__port_506344', 0.0015567826, 332, '355'), ('916235064', 'safrane_1027_gao__port_506077', 0.0013957965, 332, '355'), ('916235064', 'classe_gle_1027_gao__port_506395', 0.002197974, 332, '355'), ('916235064', '0_1027_gao__port_506094', 0.00882742, 332, '355'), ('916235064', 'ix35_1027_gao__port_506219', 0.0014615508, 332, '355'), ('916235064', 'carens_1027_gao__port_506298', 0.00088252645, 332, '355'), ('916235064', 'classe_a_1027_gao__port_506339', 0.0024713753, 332, '355'), ('916235064', 'ix20_1027_gao__port_506343', 0.0010092609, 332, '355'), ('916235064', 'note_1027_gao__port_506365', 0.0015963722, 332, '355'), ('916235064', 'a5_1027_gao__port_506200', 0.0015331057, 332, '355'), ('916235064', 'sx4_1027_gao__port_506348', 0.0014918171, 332, '355'), ('916235064', 'sandero_1027_gao__port_506198', 0.0014585878, 332, '355'), ('916235064', '3008_1027_gao__port_506385', 0.005646146, 332, '355'), ('916235064', 'q50_1027_gao__port_506239', 0.0011165293, 332, '355'), ('916235064', 'latitude_1027_gao__port_506236', 0.0008019545, 332, '355'), ('916235064', 'v40_1027_gao__port_506391', 0.0017147121, 332, '355'), ('916235064', 'xsara_1027_gao__port_506087', 0.000982284, 332, '355'), ('916235064', 'grand_c_max_1027_gao__port_506342', 0.0017959407, 332, '355'), ('916235064', 'swift_1027_gao__port_506149', 0.0015021851, 332, '355'), ('916235064', 'serie_1_1027_gao__port_506184', 0.0015139596, 332, '355'), ('916235064', 'xc70_1027_gao__port_506393', 0.003619506, 332, '355'), ('916235064', 'master_1027_gao__port_506203', 0.0079577975, 332, '355'), ('916235064', 'clio_1027_gao__port_506280', 0.0029573783, 332, '355'), ('916235064', 'duster_1027_gao__port_506216', 0.00074439385, 332, '355'), ('916235064', 'traveller_1027_gao__port_506403', 0.0042947675, 332, '355'), ('916235064', 'tipo_1027_gao__port_506355', 0.0010930618, 332, '355'), ('916235064', 'rav_4_1027_gao__port_506332', 0.0013604122, 332, '355'), ('916235064', 'coccinelle_1027_gao__port_506259', 0.0034948306, 332, '355'), ('916235064', 'spacetourer_1027_gao__port_506401', 0.0030977007, 332, '355'), ('916235064', 'xe_1027_gao__port_506357', 0.0014471813, 332, '355'), ('916235064', 'ds3_1027_gao__port_506324', 0.0013094175, 332, '355'), ('916235064', 'mx_5_1027_gao__port_506098', 0.0025888614, 332, '355'), ('916235064', 'land_cruiser_1027_gao__port_506315', 0.009531014, 332, '355'), ('916235064', 'classe_b_1027_gao__port_506335', 0.0017215814, 332, '355'), ('916235064', '806_1027_gao__port_506088', 0.0025617722, 332, '355'), ('916235064', 'rx_8_1027_gao__port_506046', 0.0036217377, 332, '355'), ('916235064', 'spark_1027_gao__port_506185', 0.0010078156, 332, '355'), ('916235064', '6_1027_gao__port_506171', 0.0011182607, 332, '355'), ('916235064', 'bravo_1027_gao__port_506080', 0.0014650495, 332, '355'), ('916235064', 'nx_1027_gao__port_506345', 0.0013683324, 332, '355'), ('916235064', 'sharan_1027_gao__port_506347', 0.0050928053, 332, '355'), ('916235064', 'x_type_1027_gao__port_506067', 0.00078031374, 332, '355'), ('916235064', 'jimny_1027_gao__port_506233', 0.0046058013, 332, '355'), ('916235064', 'wrangler_1027_gao__port_506225', 0.0017998063, 332, '355'), ('916235064', 'c_crosser_1027_gao__port_506312', 0.0015927488, 332, '355'), ('916235064', 'v70_1027_gao__port_506278', 0.001967734, 332, '355'), ('916235064', 'classe_e_1027_gao__port_506300', 0.001736883, 332, '355'), ('916235064', 'classe_v_1027_gao__port_506258', 0.012731153, 332, '355'), ('916235064', 'm3_1027_gao__port_506182', 0.0023371126, 332, '355'), ('916235064', 'abarth_500_1027_gao__port_506226', 0.0040434864, 332, '355'), ('916235064', 'serie_6_1027_gao__port_506262', 0.0011315456, 332, '355'), ('916235064', 'modus_1027_gao__port_506146', 0.0018293501, 332, '355'), ('916235064', '3_1027_gao__port_506113', 0.0015082716, 332, '355'), ('916235064', '405_1027_gao__port_506108', 0.008054141, 332, '355'), ('916235064', 'allroad_1027_gao__port_506297', 0.0010597493, 332, '355'), ('916235064', 'auris_1027_gao__port_506322', 0.0011525638, 332, '355'), ('916235064', 'galaxy_1027_gao__port_506143', 0.0025148236, 332, '355'), ('916235064', 'giulietta_1027_gao__port_506363', 0.00086517376, 332, '355'), ('916235064', '106_1027_gao__port_506073', 0.008270215, 332, '355'), ('916235064', 'classe_m_1027_gao__port_506154', 0.0030016378, 332, '355'), ('916235064', 'espace_1027_gao__port_506313', 0.0010646442, 332, '355'), ('916235064', 'panda_1027_gao__port_506189', 0.009029527, 332, '355'), ('916235064', 'rcz_1027_gao__port_506197', 0.0011293838, 332, '355'), ('916235064', '4007_1027_gao__port_506162', 0.00067929964, 332, '355'), ('916235064', 'classe_cl_1027_gao__port_506249', 0.0010859962, 332, '355'), ('916235064', 'leaf_1027_gao__port_506139', 0.0018040473, 332, '355'), ('916235064', 'octavia_1027_gao__port_506237', 0.0018602833, 332, '355'), ('916235064', 'ds4_1027_gao__port_506336', 0.0024161162, 332, '355'), ('916235064', 'freelander_1027_gao__port_506084', 0.002347203, 332, '355'), ('916235064', 'evasion_1027_gao__port_506109', 0.0031138617, 332, '355'), ('916235064', 'punto_1027_gao__port_506106', 0.0019495747, 332, '355'), ('916235064', '2cv_1027_gao__port_506045', 0.007972709, 332, '355'), ('916235064', 'x4_1027_gao__port_506392', 0.0017950318, 332, '355'), ('916235064', 'antara_1027_gao__port_506247', 0.0012469103, 332, '355'), ('916235064', 'murano_1027_gao__port_506316', 0.0006088825, 332, '355'), ('916235064', 'alto_1027_gao__port_506201', 0.009231213, 332, '355'), ('916235064', 'meriva_1027_gao__port_506353', 0.0013764651, 332, '355'), ('916235064', 'orlando_1027_gao__port_506305', 0.0018462572, 332, '355'), ('916235064', 'new_beetle_1027_gao__port_506050', 0.0011638078, 332, '355'), ('916235064', '306_1027_gao__port_506145', 0.0035069382, 332, '355'), ('916235064', 'tiguan_1027_gao__port_506362', 0.0026828034, 332, '355'), ('916235064', 's_type_1027_gao__port_506101', 0.0011381725, 332, '355'), ('916235064', 'c1_1027_gao__port_506128', 0.0027516969, 332, '355'), ('916235064', 'vectra_1027_gao__port_506044', 0.001198914, 332, '355'), ('916235064', 'outlander_1027_gao__port_506317', 0.0017122364, 332, '355'), ('916235064', '307_1027_gao__port_506074', 0.0020011899, 332, '355'), ('916235064', 'a6_s6_1027_gao__port_506134', 0.0016569574, 332, '355'), ('916235064', 'nemo_combi_1027_gao__port_506196', 0.002266361, 332, '355'), ('916235064', 'berlingo_1027_gao__port_506194', 0.00466242, 332, '355'), ('916235064', 'partner_1027_gao__port_506285', 0.0039411234, 332, '355'), ('916235064', 'cayenne_1027_gao__port_506177', 0.003797953, 332, '355'), ('916235064', 'quattroporte_1027_gao__port_506240', 0.0024440042, 332, '355'), ('916235064', 'c_max_1027_gao__port_506282', 0.0013125244, 332, '355'), ('916235064', 'fabia_1027_gao__port_506396', 0.00529976, 332, '355'), ('916235064', 'cx_3_1027_gao__port_506281', 0.0014462613, 332, '355'), ('916235064', 'x_trail_1027_gao__port_506264', 0.001831579, 332, '355'), ('916235064', 'scirocco_1027_gao__port_506276', 0.0047910516, 332, '355'), ('916235064', 'matiz_1027_gao__port_506144', 0.0017560531, 332, '355'), ('916235064', 'tigra_1027_gao__port_506069', 0.00085427274, 332, '355'), ('916235064', 'escort_1027_gao__port_506091', 0.0048399456, 332, '355'), ('916235064', 'c2_1027_gao__port_506081', 0.0014903741, 332, '355'), ('916235064', 'mini_1027_gao__port_506168', 0.0011922233, 332, '355'), ('916235064', 'i30_1027_gao__port_506291', 0.0006326041, 332, '355'), ('916235064', 'picanto_1027_gao__port_506238', 0.002975102, 332, '355'), ('916235064', 'mito_1027_gao__port_506072', 0.0015078634, 332, '355'), ('916235064', 'impreza_1027_gao__port_506085', 0.002019126, 332, '355'), ('916235064', 'kangoo_1027_gao__port_506235', 0.0065837437, 332, '355'), ('916235064', 'a4_1027_gao__port_506193', 0.0019874691, 332, '355'), ('916235064', 'cayman_1027_gao__port_506268', 0.0018140579, 332, '355'), ('916235064', 'sportage_1027_gao__port_506148', 0.0014276699, 332, '355'), ('916235064', 'up_1027_gao__port_506356', 0.0068638916, 332, '355'), ('916235064', 'optima_1027_gao__port_506386', 0.00089184387, 332, '355'), ('916235064', 'defender_1027_gao__port_506229', 0.0067220153, 332, '355'), ('916235064', 'serie_2_1027_gao__port_506256', 0.0022275678, 332, '355'), ('916235064', 'edge_1027_gao__port_506187', 0.00087491586, 332, '355'), ('916235064', 'r19_1027_gao__port_506110', 0.0049420097, 332, '355'), ('916235064', 'jetta_1027_gao__port_506304', 0.003619395, 332, '355'), ('916235064', 'eos_1027_gao__port_506115', 0.003892667, 332, '355'), ('916235064', 'accord_1027_gao__port_506214', 0.0020126605, 332, '355'), ('916235064', 'yaris_1027_gao__port_506334', 0.0032334365, 332, '355'), ('916235064', 'classe_cls_1027_gao__port_506289', 0.00078520743, 332, '355'), ('916235064', 'polo_1027_gao__port_506361', 0.004311169, 332, '355'), ('916235064', 'serie_4_1027_gao__port_506366', 0.0011475736, 332, '355'), ('916235064', 'mini_cabriolet_1027_gao__port_506204', 0.00083788787, 332, '355'), ('916235064', 'prius_1027_gao__port_506190', 0.0011489936, 332, '355'), ('916235064', 'lodgy_1027_gao__port_506188', 0.0020174272, 332, '355'), ('916235064', 'serie_7_1027_gao__port_506307', 0.0012476717, 332, '355'), ('916235064', 'c15_1027_gao__port_506055', 0.017708655, 332, '355'), ('916235064', 'kadjar_1027_gao__port_506389', 0.0012505435, 332, '355'), ('916235064', 'insignia_1027_gao__port_506364', 0.001643522, 332, '355'), ('916235064', '308_1027_gao__port_506279', 0.0021238741, 332, '355'), ('916235064', 'roomster_1027_gao__port_506241', 0.0018015342, 332, '355'), ('916235064', '80_1027_gao__port_506057', 0.004613164, 332, '355'), ('916235064', '309_1027_gao__port_506063', 0.013523385, 332, '355'), ('916235064', 'tucson_1027_gao__port_506320', 0.0021236949, 332, '355'), ('916235064', 'x3_1027_gao__port_506212', 0.00089772267, 332, '355'), ('916235064', 'xf_1027_gao__port_506263', 0.0011167311, 332, '355'), ('916235064', '2008_1027_gao__port_506394', 0.0026412406, 332, '355'), ('916235064', 'passat_1027_gao__port_506306', 0.001497768, 332, '355'), ('916235064', 'compass_1027_gao__port_506260', 0.0032557587, 332, '355'), ('916235064', 'twingo_1027_gao__port_506309', 0.0064993445, 332, '355'), ('916235064', 'micra_1027_gao__port_506221', 0.0035859111, 332, '355'), ('916235064', 'golf_1027_gao__port_506155', 0.0031933535, 332, '355'), ('916235064', 'soul_1027_gao__port_506176', 0.0012890637, 332, '355'), ('916235064', 'rapid_1027_gao__port_506398', 0.002591685, 332, '355'), ('916235064', 'forester_1027_gao__port_506360', 0.0022761086, 332, '355'), ('916235064', 'slk_1027_gao__port_506210', 0.0015845188, 332, '355'), ('916235064', 'forfour_1027_gao__port_506341', 0.0021797586, 332, '355'), ('916235064', 'serie_5_1027_gao__port_506209', 0.0013695785, 332, '355'), ('916235064', 'xj_1027_gao__port_506170', 0.0026006347, 332, '355'), ('916235064', 'pajero_1027_gao__port_506097', 0.005210687, 332, '355'), ('916235064', 'agila_1027_gao__port_506119', 0.004849805, 332, '355'), ('916235064', 'a6_1027_gao__port_506163', 0.0018910874, 332, '355'), ('916235064', 'fox_1027_gao__port_506092', 0.0008465247, 332, '355'), ('916235064', 'boxster_1027_gao__port_506267', 0.0015942035, 332, '355'), ('916235064', 'altea_1027_gao__port_506246', 0.002139434, 332, '355'), ('916235064', 'samurai_1027_gao__port_506047', 0.0062512555, 332, '355'), ('916235064', 'trax_1027_gao__port_506296', 0.0019440493, 332, '355'), ('916235064', 'getz_1027_gao__port_506058', 0.0016386055, 332, '355'), ('916235064', 'cherokee_1027_gao__port_506269', 0.0029806679, 332, '355'), ('916235064', 'koleos_1027_gao__port_506378', 0.001559157, 332, '355'), ('916235064', 'z_series_1027_gao__port_506123', 0.0016563875, 332, '355'), ('916235064', 'ecosport_1027_gao__port_506271', 0.0013231306, 332, '355'), ('916235064', 'space_star_1027_gao__port_506277', 0.0021145823, 332, '355'), ('916235064', 'rs3_sportback_1027_gao__port_506207', 0.0019117694, 332, '355'), ('916235064', 'civic_1027_gao__port_506141', 0.0026899525, 332, '355'), ('916235064', 'talisman_1027_gao__port_506390', 0.00076137757, 332, '355'), ('916235064', 'f_pace_1027_gao__port_506314', 0.0016166167, 332, '355'), ('916235064', 'classe_c_1027_gao__port_506299', 0.0017941128, 332, '355'), ('916235064', 'tt_1027_gao__port_506075', 0.0013935333, 332, '355'), ('916235064', 'pathfinder_1027_gao__port_506183', 0.0016514312, 332, '355'), ('916235064', '156_1027_gao__port_506157', 0.0015443068, 332, '355'), ('916235064', 'cx_5_1027_gao__port_506228', 0.001441407, 332, '355'), ('916235064', 'scenic_1027_gao__port_506255', 0.0016084842, 332, '355'), ('916235064', 'yeti_1027_gao__port_506358', 0.0020917961, 332, '355'), ('916235064', 'mustang_1027_gao__port_506053', 0.0100493, 332, '355'), ('916235064', 'stilo_1027_gao__port_506060', 0.0010831156, 332, '355'), ('916235064', 'ateca_1027_gao__port_506382', 0.0017012578, 332, '355'), ('916235064', 'fiorino_1027_gao__port_506217', 0.009198406, 332, '355'), ('916235064', 'classe_glk_1027_gao__port_506290', 0.0017016893, 332, '355'), ('916235064', 'fortwo_1027_gao__port_506230', 0.0016011024, 332, '355'), ('916235064', 'cruze_1027_gao__port_506186', 0.0010052931, 332, '355'), ('916235064', '107_1027_gao__port_506213', 0.0016277287, 332, '355'), ('916235064', 'aygo_1027_gao__port_506248', 0.0032435243, 332, '355'), ('916235064', 'rx_1027_gao__port_506354', 0.0010632436, 332, '355'), ('916235064', '500_1027_gao__port_506245', 0.0016355375, 332, '355'), ('916235064', 'bora_1027_gao__port_506104', 0.0038163455, 332, '355'), ('916235064', 'transit_1027_gao__port_506111', 0.0048625823, 332, '355'), ('916235064', 'pt_cruiser_1027_gao__port_506054', 0.0019164293, 332, '355'), ('916235064', 'patrol_1027_gao__port_506068', 0.00423982, 332, '355'), ('916235064', 'r8_1027_gao__port_506156', 0.0012722086, 332, '355'), ('916235064', 'xm_1027_gao__port_506102', 0.0022677255, 332, '355'), ('916235064', 's60_1027_gao__port_506191', 0.0031993329, 332, '355'), ('916235064', 'aveo_1027_gao__port_506158', 0.0038403883, 332, '355'), ('916235064', 'captiva_1027_gao__port_506159', 0.0017189491, 332, '355'), ('916235064', 'ax_1027_gao__port_506153', 0.0068959156, 332, '355'), ('916235064', 'rexton_1027_gao__port_506107', 0.0013020906, 332, '355'), ('916235064', 'camaro_1027_gao__port_506056', 0.002490115, 332, '355'), ('916235064', 'ypsilon_1027_gao__port_506131', 0.0019543513, 332, '355'), ('916235064', 'delta_1027_gao__port_506165', 0.0014002442, 332, '355'), ('916235064', 'c4_1027_gao__port_506370', 0.001300966, 332, '355'), ('916235064', 'zx_1027_gao__port_506161', 0.0045934133, 332, '355'), ('916235064', 'verso_1027_gao__port_506242', 0.00077215454, 332, '355'), ('916235064', 'superb_1027_gao__port_506327', 0.0019941907, 332, '355'), ('916235064', 'r5_1027_gao__port_506253', 0.009545046, 332, '355'), ('916235064', 'caddy_1027_gao__port_506330', 0.013826187, 332, '355'), ('916235064', 'x5_1027_gao__port_506243', 0.0011203982, 332, '355'), ('916235064', 'f_type_1027_gao__port_506231', 0.0008300042, 332, '355'), ('916235064', 'fusion_1027_gao__port_506096', 0.001267059, 332, '355'), ('916235064', 'dokker_1027_gao__port_506331', 0.0053573176, 332, '355'), ('916235064', '205_1027_gao__port_506062', 0.0066842698, 332, '355'), ('916235064', 'macan_1027_gao__port_506195', 0.0015571814, 332, '355'), ('916235064', 'tourneo_1027_gao__port_506369', 0.0064016734, 332, '355'), ('916235064', '108_1027_gao__port_506384', 0.0052643456, 332, '355'), ('916235064', '9_3_1027_gao__port_506071', 0.000837467, 332, '355'), ('916235064', 'mondeo_1027_gao__port_506116', 0.0014396003, 332, '355'), ('916235064', 'cr_v_1027_gao__port_506164', 0.001641436, 332, '355'), ('916235064', 'c30_1027_gao__port_506090', 0.0017486886, 332, '355'), ('916235064', 'pulsar_1027_gao__port_506397', 0.0012019982, 332, '355'), ('916235064', 'ibiza_1027_gao__port_506273', 0.0037230975, 332, '355'), ('916235064', 'a1_1027_gao__port_506338', 0.0012346996, 332, '355'), ('916235064', 'matrix_1027_gao__port_506140', 0.000707754, 332, '355'), ('916235064', 'carnival_1027_gao__port_506136', 0.0022814134, 332, '355'), ('916235064', 'xantia_1027_gao__port_506086', 0.0021958873, 332, '355'), ('916235064', 'terrano_1027_gao__port_506083', 0.0020294124, 332, '355'), ('916235064', 'q3_1027_gao__port_506275', 0.00112656, 332, '355'), ('916235064', 'hr_v_1027_gao__port_506283', 0.0017806026, 332, '355'), ('916235064', 'expert_1027_gao__port_506142', 0.0073699327, 332, '355'), ('916235064', 'multivan_1027_gao__port_506383', 0.0065051923, 332, '355'), ('916235064', 'venga_1027_gao__port_506380', 0.00080045644, 332, '355'), ('916235064', 'scudo_1027_gao__port_506129', 0.0055925893, 332, '355'), ('916235064', 'laguna_1027_gao__port_506368', 0.00071345636, 332, '355'), ('916235064', 'vel_satis_1027_gao__port_506130', 0.0027263074, 332, '355'), ('916235064', 'b_max_1027_gao__port_506367', 0.0017247793, 332, '355'), ('916235064', 'ignis_1027_gao__port_506292', 0.004356073, 332, '355'), ('916235064', '159_1027_gao__port_506064', 0.0010782304, 332, '355'), ('916235064', 'grande_punto_1027_gao__port_506138', 0.0023635076, 332, '355'), ('916235064', 'logan_1027_gao__port_506167', 0.0043969876, 332, '355'), ('916235064', 's_max_1027_gao__port_506223', 0.001252798, 332, '355'), ('916235064', 'caravelle_1027_gao__port_506351', 0.0030298973, 332, '355'), ('916235064', 'adam_1027_gao__port_506079', 0.0010538667, 332, '355'), ('916235064', '406_1027_gao__port_506132', 0.0013574945, 332, '355'), ('916235064', 'q30_1027_gao__port_506293', 0.0009715666, 332, '355'), ('916235064', 'almera_1027_gao__port_506089', 0.0010240269, 332, '355'), ('916235064', 'corsa_1027_gao__port_506095', 0.0025204169, 332, '355'), ('916235064', 'corolla_1027_gao__port_506120', 0.0026820586, 332, '355'), ('916235064', 'xc60_1027_gao__port_506388', 0.0018987542, 332, '355'), ('916235064', 'viano_1027_gao__port_506211', 0.0026944615, 332, '355'), ('916235064', 'pro_cee_d_1027_gao__port_506274', 0.00083205686, 332, '355'), ('916235064', 'a3_1027_gao__port_506321', 0.0037381095, 332, '355'), ('916235064', 'v50_1027_gao__port_506150', 0.00079203956, 332, '355'), ('916235064', 'voyager_1027_gao__port_506169', 0.00305262, 332, '355'), ('916235064', 'corvette_1027_gao__port_506049', 0.0037230067, 332, '355'), ('916235064', 'rio_1027_gao__port_506379', 0.0017741272, 332, '355'), ('916235064', 'jazz_1027_gao__port_506252', 0.0015304898, 332, '355'), ('916235064', '200_1027_gao__port_506112', 0.004086734, 332, '355'), ('916235064', 'tts_1027_gao__port_506199', 0.0011862081, 332, '355'), ('916235064', 'zafira_1027_gao__port_506287', 0.0026951905, 332, '355'), ('916235064', 'asx_1027_gao__port_506266', 0.0011407682, 332, '355'), ('916235064', '607_1027_gao__port_506118', 0.0012529618, 332, '355'), ('916235064', '207_1027_gao__port_506103', 0.001514927, 332, '355'), ('916235064', 'classe_s_1027_gao__port_506301', 0.003165344, 332, '355'), ('916235064', 'c6_1027_gao__port_506105', 0.0017347619, 332, '355'), ('916235064', 'express_1027_gao__port_506137', 0.016725307, 332, '355'), ('916235064', 'classe_gla_1027_gao__port_506352', 0.0018256088, 332, '355'), ('916235064', 'v60_1027_gao__port_506333', 0.0021461248, 332, '355'), ('916235064', 'ka_1027_gao__port_506180', 0.0014152853, 332, '355'), ('916235064', 'range_rover_1027_gao__port_506254', 0.0020552794, 332, '355'), ('916235064', 'discovery_1027_gao__port_506375', 0.0022964561, 332, '355'), ('916235064', 'classe_r_1027_gao__port_506270', 0.0013942857, 332, '355'), ('916235064', 'transporter_1027_gao__port_506319', 0.0119690085, 332, '355'), ('916235064', 'cee_d_1027_gao__port_506288', 0.0010549326, 332, '355'), ('916235064', 'zoe_1027_gao__port_506244', 0.002071646, 332, '355'), ('916235064', 'i20_1027_gao__port_506284', 0.001786984, 332, '355'), ('916235064', 'gtv_1027_gao__port_506059', 0.005722453, 332, '355'), ('916235064', 's4_avant_1027_gao__port_506261', 0.0027664825, 332, '355'), ('916235064', 'x1_1027_gao__port_506372', 0.0017145028, 332, '355'), ('916235064', 'autres_1027_gao__port_506127', 0.0048249103, 332, '355'), ('916235064', '208_1027_gao__port_506359', 0.0018688275, 332, '355'), ('916235064', 'c8_1027_gao__port_506135', 0.0012580217, 332, '355'), ('916235064', 'astra_1027_gao__port_506215', 0.0012624713, 332, '355'), ('916235064', '2_1027_gao__port_506151', 0.0009244882, 332, '355'), ('916235064', 'doblo_1027_gao__port_506251', 0.007466171, 332, '355'), ('916235064', '807_1027_gao__port_506152', 0.0007290672, 332, '355'), ('916235064', '206_1027_gao__port_506126', 0.0010385977, 332, '355'), ('916235064', 'a7_1027_gao__port_506373', 0.0006911822, 332, '355'), ('916235064', 'renegade_1027_gao__port_506346', 0.002141996, 332, '355')]]} begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 5.9604644775390625e-06 save missing photos in datou_result : time spend for datou_step_exec : 47.20616602897644 time spend to save output : 8.025143146514893 total time spend for step 1 : 55.23130917549133 step2:argmax Thu Apr 3 06:37:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1743654993_1794822_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1743654993_1794822_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step Argmax ! calculate argmax for thcl : 355 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True photo_id : 916235064 output[photo_id] : [('916235064', 'c15_1027_gao__port_506055', 0.017708655, 332, '355'), 'temp/1743654993_1794822_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg'] begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 1 insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) first line : ('916235064', '2049863950', '332') ... last line : ('916235064', '2049863950', '332') time used for this insertion : 0.011868476867675781 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 1 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 0.015958547592163086 len list_finale : 1, len picture : 1 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('2', None, '916235064', 'c15_1027_gao__port_506055', None, None, '2049863950', '0.017708655', None)] time used for this insertion : 0.015889644622802734 saving photo_ids in datou_result photo id not in port begin to insert list_values into mtr_datou_result : length of list_values in save_final : 0 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [] time used for this insertion : 3.337860107421875e-06 save missing photos in datou_result : time spend for datou_step_exec : 0.0002219676971435547 time spend to save output : 0.04397225379943848 total time spend for step 2 : 0.04419422149658203 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : {'916235064': [('916235064', 'c15_1027_gao__port_506055', 0.017708655, 332, '355'), 'temp/1743654993_1794822_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg']} ############################### TEST tfhub2 ################################ TEST TFHUB2 ######################## test with use_multi_inputs=0 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4567 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4567 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4567 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4567 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12835 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12836 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171252784,1171252764,1171252487) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171252487 begin to download photo : 1171252764 begin to download photo : 1171252784 download finish for photo 1171252487 download finish for photo 1171252764 download finish for photo 1171252784 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.20319628715515137 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Thu Apr 3 06:37:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1743655049_1794822_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg': 1171252487, 'temp/1743655049_1794822_1171252764_29d5179a892cc50aadc9d67245534b59.jpg': 1171252764, 'temp/1743655049_1794822_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg': 1171252784} map_photo_id_path_extension : {1171252487: {'path': 'temp/1743655049_1794822_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg', 'extension': 'jpg'}, 1171252764: {'path': 'temp/1743655049_1794822_1171252764_29d5179a892cc50aadc9d67245534b59.jpg', 'extension': 'jpg'}, 1171252784: {'path': 'temp/1743655049_1794822_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3609': 1} we are using the classfication for only one thcl 3609 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1406 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1406 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1406 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1406 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1406 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1406 wait 20 seconds 2025-04-03 06:39:37.942407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-04-03 06:39:37.943141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-03 06:39:37.943243: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:39:37.943311: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:39:37.945621: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-03 06:39:37.945725: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-03 06:39:37.948370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-03 06:39:37.949511: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-03 06:39:37.954588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:39:37.955795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-03 06:39:37.956351: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-04-03 06:39:37.987241: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-04-03 06:39:37.989522: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb71c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-04-03 06:39:37.989573: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-04-03 06:39:38.009041: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x97200c80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-04-03 06:39:38.009066: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-04-03 06:39:38.009879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-03 06:39:38.010016: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:39:38.010041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-03 06:39:38.010160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-03 06:39:38.010192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-03 06:39:38.010232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-03 06:39:38.010275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-03 06:39:38.010320: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-03 06:39:38.011271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-03 06:39:38.011329: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-03 06:39:38.011371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-04-03 06:39:38.011382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-04-03 06:39:38.011392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-04-03 06:39:38.012388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 1406 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3609 To do loadFromThcl(), then load ParamDescType : thcl3609 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3609) thcls : [{'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'}] thcl {'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'} Update svm_hashtag_type_desc : 5832 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5832) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5832, 'tfhub_19_06_2023', 1280, 1280, 'tfhub_19_06_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 6, 19, 12, 55, 22), datetime.datetime(2023, 6, 19, 12, 55, 22)) model_name : tfhub_19_06_2023 model_param file didn't exist model_name : tfhub_19_06_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-04-03 06:39:46.036050: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-04-03 06:39:46.036654: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-04-03 06:39:46.037215: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-04-03 06:39:46.037780: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-04-03 06:39:46.038343: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.98G (2129957120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-04-03 06:39:46.038868: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.79G (1916961536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-04-03 06:39:46.039422: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.61G (1725265408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/tfhub_19_06_2023 /data/models_weight/tfhub_19_06_2023/Confusion_Matrix.png size_local : 57753 size in s3 : 57753 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_jrm.jpg size_local : 79724 size in s3 : 79724 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcm.jpg size_local : 83556 size in s3 : 83556 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcnc.jpg size_local : 74107 size in s3 : 74107 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pehd.jpg size_local : 72705 size in s3 : 72705 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_tapis_vide.jpg size_local : 70874 size in s3 : 70874 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 checkpoint already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216488 size in s3 : 216488 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:21 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_weights.h5 size_local : 16499144 size in s3 : 16499144 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:15 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 77, in create_tfhub_model hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module"), File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171252487, 1171252764, 1171252784] map_info['map_portfolio_photo'] : {} final : True mtd_id 4567 list_pids : [1171252487, 1171252764, 1171252784] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4567', None, '1171252487', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252764', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252784', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.014995336532592773 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 137.43048644065857 time spend to save output : 0.020802021026611328 total time spend for step 0 : 137.45128846168518 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171252784': [(1171252784, 'jrm', 0.9677492, 4674, '3609'), 'temp/1687511175_1882837_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg'], '1171252764': [(1171252764, 'jrm', 0.9853587, 4674, '3609'), 'temp/1687511175_1882837_1171252764_29d5179a892cc50aadc9d67245534b59.jpg'], '1171252487': [(1171252487, 'jrm', 0.9262757, 4674, '3609'), 'temp/1687511175_1882837_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg']} got : None ######################## test with use_multi_inputs=1 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4621 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4621 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4621 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4621 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12927 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12928 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171291875,1171275372,1171275314) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171275314 begin to download photo : 1171275372 begin to download photo : 1171291875 download finish for photo 1171275372 download finish for photo 1171275314 download finish for photo 1171291875 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.19739890098571777 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Thu Apr 3 06:39:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1743655186_1794822_1171275372_76d81364ff7df843bff095f45c07ba35.jpg': 1171275372, 'temp/1743655186_1794822_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg': 1171275314, 'temp/1743655186_1794822_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg': 1171291875} map_photo_id_path_extension : {1171275372: {'path': 'temp/1743655186_1794822_1171275372_76d81364ff7df843bff095f45c07ba35.jpg', 'extension': 'jpg'}, 1171275314: {'path': 'temp/1743655186_1794822_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg', 'extension': 'jpg'}, 1171291875: {'path': 'temp/1743655186_1794822_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3655': 1} we are using the classfication for only one thcl 3655 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 21 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 21 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 21 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 21 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 21 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 21 wait 20 seconds l 3637 free memory gpu now : 21 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3655 To do loadFromThcl(), then load ParamDescType : thcl3655 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3655) thcls : [{'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'}] thcl {'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'} Update svm_hashtag_type_desc : 5862 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5862) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5862, 'tfhub_18_7_2023', 1280, 1280, 'tfhub_18_7_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 7, 18, 22, 46, 29), datetime.datetime(2023, 7, 18, 22, 46, 29)) model_name : tfhub_18_7_2023 model_param file didn't exist model_name : tfhub_18_7_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/tfhub_18_7_2023 /data/models_weight/tfhub_18_7_2023/Confusion_Matrix.png size_local : 54360 size in s3 : 54360 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:28 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_jrm.jpg size_local : 72583 size in s3 : 72583 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcm.jpg size_local : 81681 size in s3 : 81681 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcnc.jpg size_local : 79510 size in s3 : 79510 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pehd.jpg size_local : 59936 size in s3 : 59936 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_tapis_vide.jpg size_local : 78974 size in s3 : 78974 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 checkpoint already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216529 size in s3 : 216529 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279748 size in s3 : 32279748 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_weights.h5 size_local : 16500868 size in s3 : 16500868 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:18 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 62, in create_tfhub_model fe_layer = hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module", File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171275372, 1171275314, 1171291875] map_info['map_portfolio_photo'] : {} final : True mtd_id 4621 list_pids : [1171275372, 1171275314, 1171291875] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4621', None, '1171275372', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171275314', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171291875', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.01670384407043457 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 135.0487298965454 time spend to save output : 0.017437219619750977 total time spend for step 0 : 135.06616711616516 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171291875': [(1171291875, 'tapis_vide', 0.97062814, 4723, '3655'), 'temp/1691745841_1143057_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg'], '1171275372': [(1171275372, 'tapis_vide', 0.9674145, 4723, '3655'), 'temp/1691745841_1143057_1171275372_76d81364ff7df843bff095f45c07ba35.jpg'], '1171275314': [(1171275314, 'tapis_vide', 0.96509415, 4723, '3655'), 'temp/1691745841_1143057_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg']} got : None ERROR tfhub2 FAILED ############################### TEST ordonner ################################ To do loadFromThcl(), then load ParamDescType : thcl358 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (358) thcls : [{'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3392 ['FirstUploadExperveo_vignette__port_505674', 'CAR_EXTERIEUR_Roue__port_503398', 'FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486', 'FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465', 'CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198', 'CAR_EXTERIEUR_Face_avant_axe_droit__port_504451', 'CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235', 'FirstUploadExperveo_vin__port_505675', 'CAR_EXTERIEUR_cote_droite__port_504108', 'CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565', 'FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201', 'cartegrise_orientation__port_505064', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217', 'CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531', 'CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218', 'CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214', 'CAR_EXTERIEUR_Angle_avant_droit__port_504087', 'FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484', 'CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563', 'CAR_EXTERIEUR_Angle_arriere_droit__port_504160', 'CAR_EXTERIEUR_arriere__port_504184', 'CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562', 'INTERIEUR_Compteur_kilometrique__port_503644', 'CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494', 'CAR_EXTERIEUR_Angle_arriere_gauche__port_504170', 'CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226', 'CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202', 'CAR_EXTERIEUR_moteur__port_503704', 'FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487', 'CAR_INTERIEUR_siege_arriere_class_1__port_506551', 'CAR_EXTERIEUR_avant__port_504146', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215', 'CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225', 'CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564', 'FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'CAR_INTERIEUR_coffre__port_503412', 'FirstUploadExperveo_rouetranche__port_505677', 'UploadPhotoImmatBest_class_1__port_505051', 'CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532', 'CAR_EXTERIEUR_angle_avant_gauche__port_504098', 'CAR_EXTERIEUR_face_avant_axe_gauche__port_504236', 'CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540', 'CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233', 'CAR_EXTERIEUR_roue_de_secour__port_503763', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199', 'CAR_EXTERIEUR_cote_gauche__port_504017', 'CAR_INTERIEUR_avant_volant_class_1__port_506503', 'CAR_INTERIEUR_avant_volant_class_2__port_506504', 'CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'] 51 SELECT hashtag_id,hashtag FROM MTRBack.hashtags where hashtag in ('FirstUploadExperveo_vignette__port_505674','CAR_EXTERIEUR_Roue__port_503398','FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486','FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485','CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465','CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198','CAR_EXTERIEUR_Face_avant_axe_droit__port_504451','CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235','FirstUploadExperveo_vin__port_505675','CAR_EXTERIEUR_cote_droite__port_504108','CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565','FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483','CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201','cartegrise_orientation__port_505064','CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217','CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531','CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218','CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214','CAR_EXTERIEUR_Angle_avant_droit__port_504087','FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484','CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563','CAR_EXTERIEUR_Angle_arriere_droit__port_504160','CAR_EXTERIEUR_arriere__port_504184','CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562','INTERIEUR_Compteur_kilometrique__port_503644','CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494','CAR_EXTERIEUR_Angle_arriere_gauche__port_504170','CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226','CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202','CAR_EXTERIEUR_moteur__port_503704','FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487','CAR_INTERIEUR_siege_arriere_class_1__port_506551','CAR_EXTERIEUR_avant__port_504146','CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215','CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225','CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564','FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482','CAR_INTERIEUR_coffre__port_503412','FirstUploadExperveo_rouetranche__port_505677','UploadPhotoImmatBest_class_1__port_505051','CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532','CAR_EXTERIEUR_angle_avant_gauche__port_504098','CAR_EXTERIEUR_face_avant_axe_gauche__port_504236','CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540','CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233','CAR_EXTERIEUR_roue_de_secour__port_503763','CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199','CAR_EXTERIEUR_cote_gauche__port_504017','CAR_INTERIEUR_avant_volant_class_1__port_506503','CAR_INTERIEUR_avant_volant_class_2__port_506504','CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'); 51 dict_keys(['cartegrise_orientation__port_505064', 'car_exterieur_angle_arriere_droit_axe_arriere__port_504217', 'car_exterieur_angle_arriere_droit_axe_droit__port_504215', 'car_exterieur_angle_arriere_droit__port_504160', 'car_exterieur_angle_arriere_gauche_axe_arriere__port_504201', 'car_exterieur_angle_arriere_gauche_axe_gauche__port_504199', 'car_exterieur_angle_arriere_gauche__port_504170', 'car_exterieur_angle_avant_droit_axe_arriere__port_504226', 'car_exterieur_angle_avant_droit_axe_droit__port_504225', 'car_exterieur_angle_avant_droit__port_504087', 'car_exterieur_angle_avant_gauche_axe_avant__port_504235', 'car_exterieur_angle_avant_gauche_axe_gauche__port_504234', 'car_exterieur_angle_avant_gauche__port_504098', 'car_exterieur_arriere__port_504184', 'car_exterieur_avant__port_504146', 'car_exterieur_cote_droite__port_504108', 'car_exterieur_cote_droit_axe_arriere__port_504214', 'car_exterieur_cote_droit_axe_avant__port_504465', 'car_exterieur_cote_gauche_axe_arriere__port_504198', 'car_exterieur_cote_gauche_axe_avant__port_504233', 'car_exterieur_cote_gauche__port_504017', 'car_exterieur_face_arriere_axe_droit__port_504218', 'car_exterieur_face_arriere_axe_gauche__port_504202', 'car_exterieur_face_avant_axe_droit__port_504451', 'car_exterieur_face_avant_axe_gauche__port_504236', 'car_exterieur_moteur__port_503704', 'car_exterieur_roue_de_secour__port_503763', 'car_exterieur_roue__port_503398', 'car_interieur_avant_volant_class_1__port_506503', 'car_interieur_avant_volant_class_2__port_506504', 'car_interieur_avant_volant_class_6_boutonrond__port_506562', 'car_interieur_avant_volant_class_6_class_2__port_506563', 'car_interieur_avant_volant_class_6_ecrangrosplan__port_506564', 'car_interieur_avant_volant_class_6_levierdevitesse__port_506565', 'car_interieur_avant_vue-arriere_class_1__port_506531', 'car_interieur_avant_vue-arriere_class_2__port_506532', 'car_interieur_avant_vue_droite_habitacle_class_1__port_506540', 'car_interieur_avant_vue_gauche_habitacle_class_1__port_506494', 'car_interieur_coffre__port_503412', 'car_interieur_siege_arriere_class_1__port_506551', 'firstuploadexperveo_carrosseriegrosplan_carrosserie__port_506483', 'firstuploadexperveo_carrosseriegrosplan_class_6__port_506487', 'firstuploadexperveo_carrosseriegrosplan_morceauderoue__port_506484', 'firstuploadexperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'firstuploadexperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'firstuploadexperveo_carrosseriegrosplan_vindanslamoquette__port_506486', 'firstuploadexperveo_rouetranche__port_505677', 'firstuploadexperveo_vignette__port_505674', 'firstuploadexperveo_vin__port_505675', 'interieur_compteur_kilometrique__port_503644', 'uploadphotoimmatbest_class_1__port_505051']) select photo_hashtag_type from MTRDatou.classification_theme where id = 358 thcl : 358 photo_hashtag_type : 337 SELECT phi.hashtag_id , phi.photo_id FROM MTRBack.photo_hashtag_ids phi, MTRUser.mtr_portfolio_photos mp where phi.type = 337 and phi.photo_id = mp.mtr_photo_id and mp.mtr_portfolio_id =510365; {510365: [(917973295, 1), (917973297, 1), (917973302, 1), (917973293, 7), (917973296, 11), (917973300, 11), (917973286, 13), (917973289, 13), (917973301, 24), (917973285, 29), (917973290, 29), (917973299, 29), (917973304, 35), (917973287, 36), (917973298, 36), (917973305, 36), (917973292, 37), (917973291, 41), (917973303, 41), (917973294, 42), (917973288, 46)]} ############################### TEST rotate ################################ test rotate only Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=230 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=230 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 230 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=230 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.17762112617492676 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:rotate Thu Apr 3 06:42:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step_rotate ! We are in a linear step without datou_depend ! rotate photos of 90,180,270 degres batch 1 select photo_id, hashtag_id, `type`, x0, x1, y0, y1, score, id from MTRPhoto.crop_hashtag_ids where photo_id in ( 917849322) and `type` in (0) Loaded 0 chid ids of type : 0 SELECT crop_hashtag_id, points FROM MTRPhoto.crop_polygon_points WHERE crop_hashtag_id in () map_chi : {} photo_id in download_rotate_and_save : 917849322 list_chi_loc : 0 Use all angle ! Rotation of photo 917849322 of 90 degree temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 90 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] 90 [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] shrink_image : False image_rotate : image_path : temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 180 degree temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 180 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] 180 [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] shrink_image : False image_rotate : image_path : temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 270 degree temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 270 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] 270 [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] shrink_image : False image_rotate : image_path : temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg image_rotate.mode : RGB About to upload 3 photos upload in portfolio : 551782 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1743655365_1794822 we have uploaded 3 photos in the portfolio 551782 time of upload the photos Elapsed time : 1.7435364723205566 map_filename_photo_id : 3 map_filename_photo_id : {'temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg': 1349732475, 'temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg': 1349732476, 'temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg': 1349732477} Len new_chis : 3 Len list_new_chi_with_photo_id : 0 of type : 0 list_new_chi_with_photo_id : [] After datou_step_exec type output : time spend for datou_step_exec : 2.0079774856567383 time spend to save output : 5.793571472167969e-05 total time spend for step 1 : 2.00803542137146 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : rotate we use saveGeneral [917849322] map_info['map_portfolio_photo'] : {} final : True mtd_id 230 list_pids : [917849322] Looping around the photos to save general results len do output : 3 /1349732475Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1349732476Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1349732477Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('230', None, None, None, None, None, None, None, None) ('230', None, '917849322', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('230', None, '1349732475', 'None', None, None, None, None, None), ('230', None, '1349732476', 'None', None, None, None, None, None), ('230', None, '1349732477', 'None', None, None, None, None, None), ('230', None, '917849322', None, None, None, None, None, None)] time used for this insertion : 0.18353676795959473 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {1349732475: ['917849322', 'temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg', []], 1349732476: ['917849322', 'temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg', []], 1349732477: ['917849322', 'temp/1743655364_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg', []]} test rotate only is a success ! test rotate conditionnel Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=233 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=233 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 233 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=233 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax, rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.15816116333007812 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 3 step1:thcl Thu Apr 3 06:42:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1743655367_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1743655367_1794822_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'500': 1} we are using the classfication for only one thcl 500 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (2448, 3264, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.00021696090698242188 time to convert the images to numpy array : 2.1906514167785645 total time to convert the images to numpy array : 2.1913914680480957 list photo_ids error: [] list photo_ids correct : [917849322] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 500 To do loadFromThcl(), then load ParamDescType : thcl500 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (500) thcls : [{'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'}] thcl {'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'} Update svm_hashtag_type_desc : 3517 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) To loadFromThcl() : net_3517 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 21 wait 20 seconds l 3637 free memory gpu now : 21 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) param : , param.caffemodel : orientation_carte_grise_all_2 None mean_file_type : mean_file_path : prototxt_file_path : model : orientation_carte_grise_all_2 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : orientation_carte_grise_all_2 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/orientation_carte_grise_all_2 /data/models_weight/orientation_carte_grise_all_2/caffemodel size_local : 537110520 size in s3 : 537110520 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:17 caffemodel already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_fc.prototxt size_local : 1130 size in s3 : 1130 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt size_local : 5653 size in s3 : 5653 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:31 mean.npy already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/synset_words.txt size_local : 159 size in s3 : 159 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt caffemodel_filename : /data/models_weight/orientation_carte_grise_all_2/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 21 wait 20 seconds WARNING: Logging before InitGoogleLogging() is written to STDERR F0403 06:43:38.046085 1794822 syncedmem.cpp:78] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 38.29user 27.12system 8:14.07elapsed 13%CPU (0avgtext+0avgdata 5409260maxresident)k 6627360inputs+22064outputs (12134major+3436149minor)pagefaults 0swaps