python /home/admin/mtr/script_for_cron.py -j coverage -m 9 -a '' -s coverage -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded root_folder /data_2/data_log/job/2025/May/26052025/coverage/ git_velours : /home/admin/workarea/git/Velours/ out_folder_name htmlcov output_folder /data_2/data_log/job/2025/May/26052025/coverage/htmlcov new path : /data_2/data_log/job/2025/May/26052025/coverage/ command : coverage3 run /home/admin/workarea/git/Velours/python/tests/python_tests.py --short_python3 `cat ~/.fotonower_pass/bdd.py.pass` cat: /home/admin/.fotonower_pass/bdd.py.pass: Aucun fichier ou dossier de ce type import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data python version used : 3 #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 7035 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.14733219146728516 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Mon May 26 11:20:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 7035 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-05-26 11:20:33.396119: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-26 11:20:33.423134: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-26 11:20:33.425262: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7440000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-26 11:20:33.425321: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-26 11:20:33.429704: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-26 11:20:33.693929: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3c9e1880 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-26 11:20:33.693988: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-26 11:20:33.694995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 11:20:33.695777: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 11:20:33.699653: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 11:20:33.702724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 11:20:33.703735: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 11:20:33.706979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 11:20:33.708543: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 11:20:33.713900: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 11:20:33.715239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 11:20:33.715325: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 11:20:33.715992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-26 11:20:33.716008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-26 11:20:33.716016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-26 11:20:33.717984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6470 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-05-26 11:20:34.688159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 11:20:34.688248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 11:20:34.688271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 11:20:34.688298: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 11:20:34.688321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 11:20:34.688339: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 11:20:34.688358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 11:20:34.688376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 11:20:34.689729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 11:20:34.691220: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 11:20:34.691274: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 11:20:34.691294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 11:20:34.691318: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 11:20:34.691340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 11:20:34.691357: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 11:20:34.691376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 11:20:34.691396: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 11:20:34.692770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 11:20:34.692814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-26 11:20:34.692825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-26 11:20:34.692834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-26 11:20:34.694324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6470 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-05-26 11:20:45.125141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 11:20:45.363608: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 11:20:47.664991: E tensorflow/stream_executor/cuda/cuda_driver.cc:910] failed to synchronize the stop event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-26 11:20:47.665072: E tensorflow/stream_executor/gpu/gpu_timer.cc:55] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-26 11:20:47.665101: E tensorflow/stream_executor/gpu/gpu_timer.cc:60] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-26 11:20:47.665144: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-26 11:20:47.665161: E tensorflow/stream_executor/stream.cc:5485] Internal: Failed to enqueue async memset operation: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-26 11:20:47.665184: W tensorflow/core/kernels/gpu_utils.cc:69] Failed to check cudnn convolutions for out-of-bounds reads and writes with an error message: 'Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered'; skipping this check. This only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-05-26 11:20:47.665198: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-26 11:20:47.665211: I tensorflow/stream_executor/stream.cc:4963] [stream=0x3e254980,impl=0x3e2537c0] did not memzero GPU location; source: 0x7f7298ff7020 local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (480, 640, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 640.00000 error in detect the image : temp/1748251229_3268756_957285035_a42482e51c93c8025d243dd179aee85b.jpg 2 root error(s) found. (0) Internal: cuDNN launch failure : input shape([1,128,80,80]) filter shape([3,3,128,128]) [[node res3a_branch2b/convolution (defined at usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:3007) ]] [[ROI/strided_slice_20/_22]] (1) Internal: cuDNN launch failure : input shape([1,128,80,80]) filter shape([3,3,128,128]) [[node res3a_branch2b/convolution (defined at usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:3007) ]] 0 successful operations. 0 derived errors ignored. [Op:__inference_keras_scratch_graph_13584] Function call stack: keras_scratch_graph -> keras_scratch_graph Detection mask done ! Trying to reset tf kernel 3269038 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 404 tf kernel not reseted sub process len(results) : 0 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5437 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] WARNING : results is empty ! time spent for convertir_results : 0.7704544067382812 time spend for datou_step_exec : 21.795522212982178 time spend to save output : 4.2438507080078125e-05 total time spend for step 1 : 21.795564651489258 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.014907360076904297 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'957285035': [[], 'temp/1748251229_3268756_957285035_a42482e51c93c8025d243dd179aee85b.jpg']} free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5145 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.14929890632629395 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Mon May 26 11:20:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 947 wait 20 seconds l 3637 free memory gpu now : 947 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-05-26 11:21:17.196699: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-26 11:21:17.223126: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-26 11:21:17.225107: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7444000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-26 11:21:17.225161: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-26 11:21:17.229651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-26 11:21:17.372420: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d4bcd60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-26 11:21:17.372468: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-26 11:21:17.373058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 11:21:17.373928: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 11:21:17.378097: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 11:21:17.382363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 11:21:17.384466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 11:21:17.389150: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 11:21:17.391383: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 11:21:17.401681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 11:21:17.403261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 11:21:17.403442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 11:21:17.404298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-26 11:21:17.404353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-26 11:21:17.404370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-26 11:21:17.405568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 176 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-05-26 11:21:17.638766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 11:21:17.638870: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 11:21:17.638893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 11:21:17.638937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 11:21:17.638960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 11:21:17.638979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 11:21:17.638998: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 11:21:17.639017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 11:21:17.639729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 11:21:17.640623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 11:21:17.640672: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 11:21:17.640692: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 11:21:17.640711: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 11:21:17.640730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 11:21:17.640749: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 11:21:17.640767: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 11:21:17.640786: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 11:21:17.641493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 11:21:17.641527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-26 11:21:17.641535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-26 11:21:17.641543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-26 11:21:17.642283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 176 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2025-05-26 11:21:30.906996: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 9.00MiB (rounded to 9437184) Current allocation summary follows. 2025-05-26 11:21:30.910122: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-05-26 11:21:30.910159: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 65, Chunks in use: 65. 16.2KiB allocated for chunks. 16.2KiB in use in bin. 8.9KiB client-requested in use in bin. 2025-05-26 11:21:30.910173: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-05-26 11:21:30.910186: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 251, Chunks in use: 251. 251.2KiB allocated for chunks. 251.2KiB in use in bin. 251.0KiB client-requested in use in bin. 2025-05-26 11:21:30.910198: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 40, Chunks in use: 40. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-05-26 11:21:30.910210: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 120, Chunks in use: 120. 487.5KiB allocated for chunks. 487.5KiB in use in bin. 480.0KiB client-requested in use in bin. 2025-05-26 11:21:30.910222: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 10, Chunks in use: 10. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-05-26 11:21:30.910234: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2025-05-26 11:21:30.910245: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-05-26 11:21:30.910257: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-05-26 11:21:30.910268: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-05-26 11:21:30.910280: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 8, Chunks in use: 7. 2.35MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2025-05-26 11:21:30.910291: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 7, Chunks in use: 6. 4.39MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2025-05-26 11:21:30.910304: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 48, Chunks in use: 45. 54.12MiB allocated for chunks. 50.62MiB in use in bin. 45.00MiB client-requested in use in bin. 2025-05-26 11:21:30.910315: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 28, Chunks in use: 25. 64.25MiB allocated for chunks. 55.75MiB in use in bin. 55.75MiB client-requested in use in bin. 2025-05-26 11:21:30.910327: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 4, Chunks in use: 2. 18.00MiB allocated for chunks. 8.00MiB in use in bin. 8.00MiB client-requested in use in bin. 2025-05-26 11:21:30.910339: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 3, Chunks in use: 2. 31.88MiB allocated for chunks. 23.88MiB in use in bin. 17.00MiB client-requested in use in bin. 2025-05-26 11:21:30.910372: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-26 11:21:30.910383: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-26 11:21:30.910393: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-26 11:21:30.910403: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-26 11:21:30.910413: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-26 11:21:30.910424: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 9.00MiB was 8.00MiB, Chunk State: 2025-05-26 11:21:30.910448: I tensorflow/core/common_runtime/bfc_allocator.cc:963] Size: 8.00MiB | Requested Size: 8.00MiB | in_use: 0 | bin_num: 15, prev: Size: 9.00MiB | Requested Size: 9.00MiB | in_use: 1 | bin_num: -1, next: Size: 14.88MiB | Requested Size: 8.00MiB | in_use: 1 | bin_num: -1 2025-05-26 11:21:30.910461: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 52297728 2025-05-26 11:21:30.910475: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7256000000 of size 4194304 next 631 2025-05-26 11:21:30.910487: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7256400000 of size 4194304 next 617 2025-05-26 11:21:30.910496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7256800000 of size 4194304 next 616 2025-05-26 11:21:30.910507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7256c00000 of size 6291456 next 608 2025-05-26 11:21:30.910516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7257200000 of size 9437184 next 607 2025-05-26 11:21:30.910525: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7257b00000 of size 8388608 next 625 2025-05-26 11:21:30.910534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7258300000 of size 15597568 next 18446744073709551615 2025-05-26 11:21:30.910543: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 67108864 2025-05-26 11:21:30.910552: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725a000000 of size 2359296 next 370 2025-05-26 11:21:30.910561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725a240000 of size 1048576 next 394 2025-05-26 11:21:30.910569: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f725a340000 of size 1310720 next 388 2025-05-26 11:21:30.910578: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725a480000 of size 2359296 next 387 2025-05-26 11:21:30.910586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725a6c0000 of size 1048576 next 412 2025-05-26 11:21:30.910596: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725a7c0000 of size 1310720 next 406 2025-05-26 11:21:30.910604: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725a900000 of size 2359296 next 405 2025-05-26 11:21:30.910613: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725ab40000 of size 1048576 next 430 2025-05-26 11:21:30.910621: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725ac40000 of size 1310720 next 424 2025-05-26 11:21:30.910630: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725ad80000 of size 2359296 next 423 2025-05-26 11:21:30.910638: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725afc0000 of size 1048576 next 447 2025-05-26 11:21:30.910656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725b0c0000 of size 1310720 next 441 2025-05-26 11:21:30.910665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725b200000 of size 2359296 next 440 2025-05-26 11:21:30.910673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725b440000 of size 1048576 next 465 2025-05-26 11:21:30.910682: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725b540000 of size 1310720 next 459 2025-05-26 11:21:30.910690: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725b680000 of size 2359296 next 458 2025-05-26 11:21:30.910698: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725b8c0000 of size 1048576 next 483 2025-05-26 11:21:30.910707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725b9c0000 of size 1310720 next 477 2025-05-26 11:21:30.910715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725bb00000 of size 2359296 next 476 2025-05-26 11:21:30.910724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725bd40000 of size 1048576 next 501 2025-05-26 11:21:30.910732: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725be40000 of size 1310720 next 495 2025-05-26 11:21:30.910740: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725bf80000 of size 2359296 next 494 2025-05-26 11:21:30.910749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725c1c0000 of size 1048576 next 519 2025-05-26 11:21:30.910758: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725c2c0000 of size 1310720 next 513 2025-05-26 11:21:30.910766: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725c400000 of size 2359296 next 512 2025-05-26 11:21:30.910774: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725c640000 of size 1048576 next 537 2025-05-26 11:21:30.910783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725c740000 of size 1310720 next 531 2025-05-26 11:21:30.910791: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725c880000 of size 2359296 next 530 2025-05-26 11:21:30.910800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725cac0000 of size 1048576 next 555 2025-05-26 11:21:30.910809: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725cbc0000 of size 1310720 next 549 2025-05-26 11:21:30.910818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725cd00000 of size 2359296 next 548 2025-05-26 11:21:30.910827: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725cf40000 of size 1048576 next 573 2025-05-26 11:21:30.910836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725d040000 of size 1310720 next 567 2025-05-26 11:21:30.910845: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725d180000 of size 2359296 next 566 2025-05-26 11:21:30.910873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725d3c0000 of size 1048576 next 591 2025-05-26 11:21:30.910882: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f725d4c0000 of size 1310720 next 585 2025-05-26 11:21:30.910891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725d600000 of size 2359296 next 584 2025-05-26 11:21:30.910919: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f725d840000 of size 2097152 next 597 2025-05-26 11:21:30.910929: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725da40000 of size 2097152 next 596 2025-05-26 11:21:30.910938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f725dc40000 of size 3932160 next 18446744073709551615 2025-05-26 11:21:30.910947: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432 2025-05-26 11:21:30.910957: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725e000000 of size 2359296 next 245 2025-05-26 11:21:30.910965: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725e240000 of size 1048576 next 269 2025-05-26 11:21:30.910992: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725e340000 of size 1310720 next 263 2025-05-26 11:21:30.911002: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725e480000 of size 2359296 next 262 2025-05-26 11:21:30.911010: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725e6c0000 of size 1048576 next 287 2025-05-26 11:21:30.911019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725e7c0000 of size 1310720 next 281 2025-05-26 11:21:30.911028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725e900000 of size 2359296 next 280 2025-05-26 11:21:30.911036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725eb40000 of size 1048576 next 304 2025-05-26 11:21:30.911045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725ec40000 of size 1310720 next 298 2025-05-26 11:21:30.911054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725ed80000 of size 2359296 next 297 2025-05-26 11:21:30.911063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725efc0000 of size 1048576 next 322 2025-05-26 11:21:30.911072: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725f0c0000 of size 1310720 next 316 2025-05-26 11:21:30.911080: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725f200000 of size 2359296 next 315 2025-05-26 11:21:30.911089: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725f440000 of size 1048576 next 340 2025-05-26 11:21:30.911098: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725f540000 of size 1310720 next 334 2025-05-26 11:21:30.911106: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725f680000 of size 2359296 next 333 2025-05-26 11:21:30.911115: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725f8c0000 of size 1048576 next 358 2025-05-26 11:21:30.911124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725f9c0000 of size 1310720 next 352 2025-05-26 11:21:30.911132: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725fb00000 of size 2359296 next 351 2025-05-26 11:21:30.911141: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725fd40000 of size 1048576 next 376 2025-05-26 11:21:30.911150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f725fe40000 of size 1835008 next 18446744073709551615 2025-05-26 11:21:30.911159: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216 2025-05-26 11:21:30.911168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7260000000 of size 2359296 next 181 2025-05-26 11:21:30.911177: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7260240000 of size 2097152 next 196 2025-05-26 11:21:30.911186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7260440000 of size 1048576 next 216 2025-05-26 11:21:30.911195: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7260540000 of size 1310720 next 209 2025-05-26 11:21:30.911204: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7260680000 of size 2359296 next 208 2025-05-26 11:21:30.911212: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72608c0000 of size 1048576 next 234 2025-05-26 11:21:30.911221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72609c0000 of size 1310720 next 228 2025-05-26 11:21:30.911230: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7260b00000 of size 2359296 next 227 2025-05-26 11:21:30.911239: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7260d40000 of size 2883584 next 18446744073709551615 2025-05-26 11:21:30.911248: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-05-26 11:21:30.911256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262600000 of size 147456 next 55 2025-05-26 11:21:30.911272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262624000 of size 65536 next 78 2025-05-26 11:21:30.911282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262634000 of size 4096 next 191 2025-05-26 11:21:30.911291: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262635000 of size 4096 next 192 2025-05-26 11:21:30.911299: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262636000 of size 4096 next 193 2025-05-26 11:21:30.911308: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262637000 of size 256 next 194 2025-05-26 11:21:30.911317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262637100 of size 256 next 195 2025-05-26 11:21:30.911325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262637200 of size 4096 next 197 2025-05-26 11:21:30.911334: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262638200 of size 4096 next 198 2025-05-26 11:21:30.911342: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262639200 of size 4096 next 199 2025-05-26 11:21:30.911351: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263a200 of size 4096 next 200 2025-05-26 11:21:30.911359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263b200 of size 4096 next 201 2025-05-26 11:21:30.911368: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263c200 of size 1024 next 202 2025-05-26 11:21:30.911377: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263c600 of size 1024 next 204 2025-05-26 11:21:30.911385: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263ca00 of size 1024 next 205 2025-05-26 11:21:30.911394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263ce00 of size 1024 next 206 2025-05-26 11:21:30.911402: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263d200 of size 1024 next 207 2025-05-26 11:21:30.911411: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263d600 of size 1024 next 210 2025-05-26 11:21:30.911419: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263da00 of size 1024 next 211 2025-05-26 11:21:30.911428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263de00 of size 1024 next 212 2025-05-26 11:21:30.911437: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263e200 of size 1024 next 213 2025-05-26 11:21:30.911445: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263e600 of size 1024 next 214 2025-05-26 11:21:30.911454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263ea00 of size 4096 next 215 2025-05-26 11:21:30.911462: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726263fa00 of size 4096 next 217 2025-05-26 11:21:30.911471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262640a00 of size 4096 next 218 2025-05-26 11:21:30.911479: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262641a00 of size 4096 next 219 2025-05-26 11:21:30.911488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262642a00 of size 4096 next 220 2025-05-26 11:21:30.911496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262643a00 of size 1024 next 221 2025-05-26 11:21:30.911504: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262643e00 of size 1024 next 223 2025-05-26 11:21:30.911513: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262644200 of size 1024 next 224 2025-05-26 11:21:30.911522: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262644600 of size 1024 next 225 2025-05-26 11:21:30.911530: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262644a00 of size 1024 next 226 2025-05-26 11:21:30.911539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262644e00 of size 1024 next 229 2025-05-26 11:21:30.911547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262645200 of size 1024 next 230 2025-05-26 11:21:30.911563: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262645600 of size 1024 next 231 2025-05-26 11:21:30.911572: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262645a00 of size 1024 next 232 2025-05-26 11:21:30.911581: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262645e00 of size 1024 next 233 2025-05-26 11:21:30.911590: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262646200 of size 7680 next 72 2025-05-26 11:21:30.911598: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262648000 of size 147456 next 71 2025-05-26 11:21:30.911608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726266c000 of size 131072 next 87 2025-05-26 11:21:30.911616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726268c000 of size 4096 next 289 2025-05-26 11:21:30.911625: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726268d000 of size 4096 next 290 2025-05-26 11:21:30.911633: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726268e000 of size 4096 next 291 2025-05-26 11:21:30.911642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726268f000 of size 1024 next 292 2025-05-26 11:21:30.911650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726268f400 of size 1024 next 293 2025-05-26 11:21:30.911659: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726268f800 of size 1024 next 294 2025-05-26 11:21:30.911667: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726268fc00 of size 1024 next 295 2025-05-26 11:21:30.911676: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262690000 of size 1024 next 296 2025-05-26 11:21:30.911684: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262690400 of size 1024 next 299 2025-05-26 11:21:30.911693: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262690800 of size 1024 next 300 2025-05-26 11:21:30.911701: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262690c00 of size 1024 next 301 2025-05-26 11:21:30.911710: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262691000 of size 1024 next 302 2025-05-26 11:21:30.911718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262691400 of size 1024 next 303 2025-05-26 11:21:30.911727: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262691800 of size 4096 next 305 2025-05-26 11:21:30.911735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262692800 of size 4096 next 306 2025-05-26 11:21:30.911744: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262693800 of size 4096 next 307 2025-05-26 11:21:30.911752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262694800 of size 4096 next 308 2025-05-26 11:21:30.911760: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262695800 of size 4096 next 309 2025-05-26 11:21:30.911769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262696800 of size 1024 next 310 2025-05-26 11:21:30.911777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262696c00 of size 1024 next 311 2025-05-26 11:21:30.911786: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262697000 of size 1024 next 312 2025-05-26 11:21:30.911794: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262697400 of size 1024 next 313 2025-05-26 11:21:30.911803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262697800 of size 1024 next 314 2025-05-26 11:21:30.911811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262697c00 of size 1024 next 317 2025-05-26 11:21:30.911820: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262698000 of size 1024 next 318 2025-05-26 11:21:30.911829: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262698400 of size 1024 next 319 2025-05-26 11:21:30.911843: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262698800 of size 1024 next 320 2025-05-26 11:21:30.911852: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262698c00 of size 1024 next 321 2025-05-26 11:21:30.911861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262699000 of size 4096 next 323 2025-05-26 11:21:30.911869: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269a000 of size 4096 next 324 2025-05-26 11:21:30.911878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269b000 of size 4096 next 325 2025-05-26 11:21:30.911886: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269c000 of size 4096 next 326 2025-05-26 11:21:30.911895: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269d000 of size 4096 next 327 2025-05-26 11:21:30.911903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269e000 of size 1024 next 328 2025-05-26 11:21:30.911912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269e400 of size 1024 next 329 2025-05-26 11:21:30.911920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269e800 of size 1024 next 330 2025-05-26 11:21:30.911928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269ec00 of size 1024 next 331 2025-05-26 11:21:30.911937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269f000 of size 1024 next 332 2025-05-26 11:21:30.911945: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269f400 of size 1024 next 335 2025-05-26 11:21:30.911954: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269f800 of size 1024 next 336 2025-05-26 11:21:30.911962: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726269fc00 of size 1024 next 337 2025-05-26 11:21:30.911971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a0000 of size 1024 next 338 2025-05-26 11:21:30.911979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a0400 of size 1024 next 339 2025-05-26 11:21:30.911987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a0800 of size 4096 next 341 2025-05-26 11:21:30.911996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a1800 of size 4096 next 342 2025-05-26 11:21:30.912005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a2800 of size 4096 next 343 2025-05-26 11:21:30.912014: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a3800 of size 4096 next 344 2025-05-26 11:21:30.912023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a4800 of size 4096 next 345 2025-05-26 11:21:30.912032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a5800 of size 1024 next 346 2025-05-26 11:21:30.912041: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a5c00 of size 1024 next 347 2025-05-26 11:21:30.912050: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a6000 of size 1024 next 348 2025-05-26 11:21:30.912059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a6400 of size 1024 next 349 2025-05-26 11:21:30.912068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a6800 of size 1024 next 350 2025-05-26 11:21:30.912077: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a6c00 of size 1024 next 353 2025-05-26 11:21:30.912086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a7000 of size 1024 next 354 2025-05-26 11:21:30.912094: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a7400 of size 1024 next 355 2025-05-26 11:21:30.912103: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a7800 of size 1024 next 356 2025-05-26 11:21:30.912112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a7c00 of size 1024 next 357 2025-05-26 11:21:30.912128: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a8000 of size 4096 next 359 2025-05-26 11:21:30.912137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626a9000 of size 4096 next 360 2025-05-26 11:21:30.912146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626aa000 of size 4096 next 361 2025-05-26 11:21:30.912154: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ab000 of size 4096 next 362 2025-05-26 11:21:30.912163: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ac000 of size 4096 next 363 2025-05-26 11:21:30.912172: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ad000 of size 1024 next 364 2025-05-26 11:21:30.912180: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ad400 of size 1024 next 365 2025-05-26 11:21:30.912190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ad800 of size 1024 next 366 2025-05-26 11:21:30.912199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626adc00 of size 1024 next 367 2025-05-26 11:21:30.912207: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ae000 of size 1024 next 368 2025-05-26 11:21:30.912216: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ae400 of size 1024 next 371 2025-05-26 11:21:30.912225: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ae800 of size 1024 next 372 2025-05-26 11:21:30.912233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626aec00 of size 1024 next 373 2025-05-26 11:21:30.912242: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626af000 of size 1024 next 374 2025-05-26 11:21:30.912251: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626af400 of size 1024 next 375 2025-05-26 11:21:30.912259: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626af800 of size 4096 next 377 2025-05-26 11:21:30.912268: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b0800 of size 4096 next 378 2025-05-26 11:21:30.912276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b1800 of size 4096 next 379 2025-05-26 11:21:30.912285: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b2800 of size 4096 next 380 2025-05-26 11:21:30.912293: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b3800 of size 4096 next 381 2025-05-26 11:21:30.912302: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b4800 of size 1024 next 382 2025-05-26 11:21:30.912311: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b4c00 of size 1024 next 383 2025-05-26 11:21:30.912319: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b5000 of size 1024 next 384 2025-05-26 11:21:30.912328: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b5400 of size 1024 next 385 2025-05-26 11:21:30.912337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b5800 of size 1024 next 386 2025-05-26 11:21:30.912345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b5c00 of size 1024 next 389 2025-05-26 11:21:30.912354: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b6000 of size 1024 next 390 2025-05-26 11:21:30.912362: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b6400 of size 1024 next 391 2025-05-26 11:21:30.912371: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b6800 of size 1024 next 392 2025-05-26 11:21:30.912379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b6c00 of size 1024 next 393 2025-05-26 11:21:30.912388: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b7000 of size 4096 next 395 2025-05-26 11:21:30.912396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b8000 of size 4096 next 396 2025-05-26 11:21:30.912411: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626b9000 of size 4096 next 397 2025-05-26 11:21:30.912420: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ba000 of size 4096 next 398 2025-05-26 11:21:30.912429: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bb000 of size 4096 next 399 2025-05-26 11:21:30.912437: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bc000 of size 1024 next 400 2025-05-26 11:21:30.912446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bc400 of size 1024 next 401 2025-05-26 11:21:30.912454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bc800 of size 1024 next 402 2025-05-26 11:21:30.912463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bcc00 of size 1024 next 403 2025-05-26 11:21:30.912471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bd000 of size 1024 next 404 2025-05-26 11:21:30.912480: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bd400 of size 1024 next 407 2025-05-26 11:21:30.912488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bd800 of size 1024 next 408 2025-05-26 11:21:30.912497: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bdc00 of size 1024 next 409 2025-05-26 11:21:30.912505: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626be000 of size 1024 next 410 2025-05-26 11:21:30.912514: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626be400 of size 1024 next 411 2025-05-26 11:21:30.912523: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626be800 of size 4096 next 413 2025-05-26 11:21:30.912531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626bf800 of size 4096 next 414 2025-05-26 11:21:30.912539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c0800 of size 4096 next 415 2025-05-26 11:21:30.912548: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c1800 of size 4096 next 416 2025-05-26 11:21:30.912556: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c2800 of size 4096 next 417 2025-05-26 11:21:30.912565: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c3800 of size 1024 next 418 2025-05-26 11:21:30.912573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c3c00 of size 1024 next 419 2025-05-26 11:21:30.912582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c4000 of size 1024 next 420 2025-05-26 11:21:30.912590: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c4400 of size 1024 next 421 2025-05-26 11:21:30.912599: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c4800 of size 1024 next 422 2025-05-26 11:21:30.912607: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c4c00 of size 1024 next 425 2025-05-26 11:21:30.912616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c5000 of size 1024 next 426 2025-05-26 11:21:30.912624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c5400 of size 1024 next 427 2025-05-26 11:21:30.912633: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c5800 of size 1024 next 428 2025-05-26 11:21:30.912641: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c5c00 of size 1024 next 429 2025-05-26 11:21:30.912650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c6000 of size 4096 next 431 2025-05-26 11:21:30.912658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c7000 of size 4096 next 432 2025-05-26 11:21:30.912667: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c8000 of size 4096 next 433 2025-05-26 11:21:30.912675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626c9000 of size 4096 next 434 2025-05-26 11:21:30.912684: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626ca000 of size 4096 next 435 2025-05-26 11:21:30.912699: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626cb000 of size 1024 next 436 2025-05-26 11:21:30.912708: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626cb400 of size 1024 next 437 2025-05-26 11:21:30.912716: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626cb800 of size 1024 next 438 2025-05-26 11:21:30.912725: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626cbc00 of size 1024 next 103 2025-05-26 11:21:30.912733: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72626cc000 of size 262144 next 102 2025-05-26 11:21:30.912742: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f726270c000 of size 999424 next 18446744073709551615 2025-05-26 11:21:30.912751: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-05-26 11:21:30.912760: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262800000 of size 589824 next 96 2025-05-26 11:21:30.912768: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262890000 of size 262144 next 117 2025-05-26 11:21:30.912777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d0000 of size 1024 next 439 2025-05-26 11:21:30.912786: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d0400 of size 1024 next 442 2025-05-26 11:21:30.912794: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d0800 of size 1024 next 443 2025-05-26 11:21:30.912803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d0c00 of size 1024 next 444 2025-05-26 11:21:30.912818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d1000 of size 1024 next 445 2025-05-26 11:21:30.912827: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d1400 of size 1024 next 446 2025-05-26 11:21:30.912836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d1800 of size 4096 next 448 2025-05-26 11:21:30.912844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d2800 of size 4096 next 449 2025-05-26 11:21:30.912853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d3800 of size 4096 next 450 2025-05-26 11:21:30.912861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d4800 of size 4096 next 451 2025-05-26 11:21:30.912870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d5800 of size 4096 next 452 2025-05-26 11:21:30.912878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d6800 of size 1024 next 453 2025-05-26 11:21:30.912886: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d6c00 of size 1024 next 454 2025-05-26 11:21:30.912895: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d7000 of size 1024 next 455 2025-05-26 11:21:30.912903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d7400 of size 1024 next 456 2025-05-26 11:21:30.912912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d7800 of size 1024 next 457 2025-05-26 11:21:30.912920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d7c00 of size 1024 next 460 2025-05-26 11:21:30.912929: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d8000 of size 1024 next 461 2025-05-26 11:21:30.912937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d8400 of size 1024 next 462 2025-05-26 11:21:30.912946: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d8800 of size 1024 next 463 2025-05-26 11:21:30.912954: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d8c00 of size 1024 next 464 2025-05-26 11:21:30.912963: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628d9000 of size 4096 next 466 2025-05-26 11:21:30.912971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628da000 of size 4096 next 467 2025-05-26 11:21:30.912987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628db000 of size 4096 next 468 2025-05-26 11:21:30.912996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628dc000 of size 4096 next 469 2025-05-26 11:21:30.913005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628dd000 of size 4096 next 470 2025-05-26 11:21:30.913013: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628de000 of size 1024 next 471 2025-05-26 11:21:30.913021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628de400 of size 1024 next 472 2025-05-26 11:21:30.913033: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628de800 of size 1024 next 473 2025-05-26 11:21:30.913042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628dec00 of size 1024 next 474 2025-05-26 11:21:30.913051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628df000 of size 1024 next 475 2025-05-26 11:21:30.913059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628df400 of size 1024 next 478 2025-05-26 11:21:30.913068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628df800 of size 1024 next 479 2025-05-26 11:21:30.913077: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628dfc00 of size 1024 next 480 2025-05-26 11:21:30.913085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e0000 of size 1024 next 481 2025-05-26 11:21:30.913094: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e0400 of size 1024 next 482 2025-05-26 11:21:30.913102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e0800 of size 4096 next 484 2025-05-26 11:21:30.913111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e1800 of size 4096 next 485 2025-05-26 11:21:30.913119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e2800 of size 4096 next 486 2025-05-26 11:21:30.913128: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e3800 of size 4096 next 487 2025-05-26 11:21:30.913136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e4800 of size 4096 next 488 2025-05-26 11:21:30.913145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e5800 of size 1024 next 489 2025-05-26 11:21:30.913153: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e5c00 of size 1024 next 490 2025-05-26 11:21:30.913162: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e6000 of size 1024 next 491 2025-05-26 11:21:30.913171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e6400 of size 1024 next 492 2025-05-26 11:21:30.913179: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e6800 of size 1024 next 493 2025-05-26 11:21:30.913188: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e6c00 of size 1024 next 496 2025-05-26 11:21:30.913197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e7000 of size 1024 next 497 2025-05-26 11:21:30.913205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e7400 of size 1024 next 498 2025-05-26 11:21:30.913214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e7800 of size 1024 next 499 2025-05-26 11:21:30.913222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e7c00 of size 1024 next 500 2025-05-26 11:21:30.913231: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e8000 of size 4096 next 502 2025-05-26 11:21:30.913240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628e9000 of size 4096 next 503 2025-05-26 11:21:30.913252: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ea000 of size 4096 next 504 2025-05-26 11:21:30.913261: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628eb000 of size 4096 next 505 2025-05-26 11:21:30.913277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ec000 of size 4096 next 506 2025-05-26 11:21:30.913287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ed000 of size 1024 next 507 2025-05-26 11:21:30.913295: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ed400 of size 1024 next 508 2025-05-26 11:21:30.913304: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ed800 of size 1024 next 509 2025-05-26 11:21:30.913313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628edc00 of size 1024 next 510 2025-05-26 11:21:30.913321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ee000 of size 1024 next 511 2025-05-26 11:21:30.913330: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ee400 of size 1024 next 514 2025-05-26 11:21:30.913339: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ee800 of size 1024 next 515 2025-05-26 11:21:30.913347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628eec00 of size 1024 next 516 2025-05-26 11:21:30.913356: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ef000 of size 1024 next 517 2025-05-26 11:21:30.913365: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ef400 of size 1024 next 518 2025-05-26 11:21:30.913373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ef800 of size 4096 next 520 2025-05-26 11:21:30.913382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f0800 of size 4096 next 521 2025-05-26 11:21:30.913391: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f1800 of size 4096 next 522 2025-05-26 11:21:30.913400: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f2800 of size 4096 next 523 2025-05-26 11:21:30.913409: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f3800 of size 4096 next 524 2025-05-26 11:21:30.913418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f4800 of size 1024 next 525 2025-05-26 11:21:30.913426: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f4c00 of size 1024 next 526 2025-05-26 11:21:30.913435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f5000 of size 1024 next 527 2025-05-26 11:21:30.913444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f5400 of size 1024 next 528 2025-05-26 11:21:30.913452: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f5800 of size 1024 next 529 2025-05-26 11:21:30.913464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f5c00 of size 1024 next 532 2025-05-26 11:21:30.913473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f6000 of size 1024 next 533 2025-05-26 11:21:30.913481: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f6400 of size 1024 next 534 2025-05-26 11:21:30.913490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f6800 of size 1024 next 535 2025-05-26 11:21:30.913500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f6c00 of size 1024 next 536 2025-05-26 11:21:30.913508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f7000 of size 4096 next 538 2025-05-26 11:21:30.913517: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f8000 of size 4096 next 539 2025-05-26 11:21:30.913526: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628f9000 of size 4096 next 540 2025-05-26 11:21:30.913534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fa000 of size 4096 next 541 2025-05-26 11:21:30.913543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fb000 of size 4096 next 542 2025-05-26 11:21:30.913551: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fc000 of size 1024 next 543 2025-05-26 11:21:30.913567: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fc400 of size 1024 next 544 2025-05-26 11:21:30.913576: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fc800 of size 1024 next 545 2025-05-26 11:21:30.913584: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fcc00 of size 1024 next 546 2025-05-26 11:21:30.913593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fd000 of size 1024 next 547 2025-05-26 11:21:30.913601: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fd400 of size 1024 next 550 2025-05-26 11:21:30.913610: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fd800 of size 1024 next 551 2025-05-26 11:21:30.913618: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fdc00 of size 1024 next 552 2025-05-26 11:21:30.913627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fe000 of size 1024 next 553 2025-05-26 11:21:30.913638: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fe400 of size 1024 next 554 2025-05-26 11:21:30.913647: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628fe800 of size 4096 next 556 2025-05-26 11:21:30.913655: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72628ff800 of size 4096 next 557 2025-05-26 11:21:30.913665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262900800 of size 4096 next 558 2025-05-26 11:21:30.913673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262901800 of size 4096 next 559 2025-05-26 11:21:30.913681: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262902800 of size 4096 next 560 2025-05-26 11:21:30.913690: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262903800 of size 1024 next 561 2025-05-26 11:21:30.913698: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262903c00 of size 1024 next 562 2025-05-26 11:21:30.913707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262904000 of size 1024 next 563 2025-05-26 11:21:30.913715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262904400 of size 1024 next 564 2025-05-26 11:21:30.913724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262904800 of size 1024 next 565 2025-05-26 11:21:30.913732: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262904c00 of size 1024 next 568 2025-05-26 11:21:30.913741: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262905000 of size 1024 next 569 2025-05-26 11:21:30.913749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262905400 of size 1024 next 570 2025-05-26 11:21:30.913758: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262905800 of size 1024 next 571 2025-05-26 11:21:30.913766: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262905c00 of size 1024 next 572 2025-05-26 11:21:30.913775: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262906000 of size 4096 next 574 2025-05-26 11:21:30.913785: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262907000 of size 4096 next 575 2025-05-26 11:21:30.913795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262908000 of size 4096 next 576 2025-05-26 11:21:30.913803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262909000 of size 4096 next 577 2025-05-26 11:21:30.913812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290a000 of size 4096 next 578 2025-05-26 11:21:30.913821: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290b000 of size 1024 next 579 2025-05-26 11:21:30.913830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290b400 of size 1024 next 580 2025-05-26 11:21:30.913839: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290b800 of size 1024 next 581 2025-05-26 11:21:30.913856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290bc00 of size 1024 next 582 2025-05-26 11:21:30.913865: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290c000 of size 1024 next 583 2025-05-26 11:21:30.913873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290c400 of size 1024 next 586 2025-05-26 11:21:30.913882: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290c800 of size 1024 next 587 2025-05-26 11:21:30.913891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290cc00 of size 1024 next 588 2025-05-26 11:21:30.913899: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290d000 of size 1024 next 589 2025-05-26 11:21:30.913908: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290d400 of size 1024 next 590 2025-05-26 11:21:30.913916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290d800 of size 4096 next 592 2025-05-26 11:21:30.913925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f726290e800 of size 6144 next 112 2025-05-26 11:21:30.913934: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262910000 of size 524288 next 111 2025-05-26 11:21:30.913943: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262990000 of size 262144 next 135 2025-05-26 11:21:30.913952: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72629d0000 of size 327680 next 124 2025-05-26 11:21:30.913961: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262a20000 of size 589824 next 123 2025-05-26 11:21:30.913970: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262ab0000 of size 262144 next 152 2025-05-26 11:21:30.913978: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262af0000 of size 327680 next 142 2025-05-26 11:21:30.913987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262b40000 of size 786432 next 18446744073709551615 2025-05-26 11:21:30.913996: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-05-26 11:21:30.914005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c00000 of size 4096 next 593 2025-05-26 11:21:30.914013: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c01000 of size 4096 next 594 2025-05-26 11:21:30.914022: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c02000 of size 4096 next 595 2025-05-26 11:21:30.914031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c03000 of size 2048 next 598 2025-05-26 11:21:30.914040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c03800 of size 2048 next 599 2025-05-26 11:21:30.914048: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c04000 of size 2048 next 600 2025-05-26 11:21:30.914057: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c04800 of size 2048 next 601 2025-05-26 11:21:30.914066: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c05000 of size 2048 next 602 2025-05-26 11:21:30.914074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c05800 of size 256 next 605 2025-05-26 11:21:30.914083: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c05900 of size 256 next 606 2025-05-26 11:21:30.914091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c05a00 of size 2048 next 604 2025-05-26 11:21:30.914100: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c06200 of size 2048 next 609 2025-05-26 11:21:30.914109: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c06a00 of size 2048 next 610 2025-05-26 11:21:30.914117: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c07200 of size 2048 next 611 2025-05-26 11:21:30.914126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c07a00 of size 2048 next 612 2025-05-26 11:21:30.914134: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c08200 of size 256 next 614 2025-05-26 11:21:30.914150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c08300 of size 256 next 615 2025-05-26 11:21:30.914159: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c08400 of size 8192 next 613 2025-05-26 11:21:30.914168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c0a400 of size 8192 next 618 2025-05-26 11:21:30.914177: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c0c400 of size 8192 next 619 2025-05-26 11:21:30.914185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c0e400 of size 8192 next 620 2025-05-26 11:21:30.914193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c10400 of size 8192 next 621 2025-05-26 11:21:30.914202: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c12400 of size 256 next 622 2025-05-26 11:21:30.914210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c12500 of size 256 next 623 2025-05-26 11:21:30.914219: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c12600 of size 8192 next 624 2025-05-26 11:21:30.914228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c14600 of size 8192 next 626 2025-05-26 11:21:30.914236: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c16600 of size 8192 next 627 2025-05-26 11:21:30.914245: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c18600 of size 8192 next 628 2025-05-26 11:21:30.914253: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c1a600 of size 8192 next 629 2025-05-26 11:21:30.914262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c1c600 of size 2048 next 630 2025-05-26 11:21:30.914271: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c1ce00 of size 2048 next 632 2025-05-26 11:21:30.914279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c1d600 of size 2048 next 633 2025-05-26 11:21:30.914288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c1de00 of size 2048 next 634 2025-05-26 11:21:30.914296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c1e600 of size 2048 next 635 2025-05-26 11:21:30.914305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7262c1ee00 of size 463360 next 160 2025-05-26 11:21:30.914313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262c90000 of size 589824 next 158 2025-05-26 11:21:30.914322: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262d20000 of size 524288 next 171 2025-05-26 11:21:30.914330: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f7262da0000 of size 1048576 next 222 2025-05-26 11:21:30.914338: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262ea0000 of size 1048576 next 190 2025-05-26 11:21:30.914347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f7262fa0000 of size 1048576 next 189 2025-05-26 11:21:30.914355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72630a0000 of size 1048576 next 203 2025-05-26 11:21:30.914363: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72631a0000 of size 1048576 next 251 2025-05-26 11:21:30.914372: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f72632a0000 of size 1441792 next 18446744073709551615 2025-05-26 11:21:30.914381: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-05-26 11:21:30.914390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00000 of size 1280 next 1 2025-05-26 11:21:30.914399: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00500 of size 256 next 5 2025-05-26 11:21:30.914408: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00600 of size 256 next 8 2025-05-26 11:21:30.914416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00700 of size 256 next 9 2025-05-26 11:21:30.914435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00800 of size 256 next 10 2025-05-26 11:21:30.914446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00900 of size 256 next 11 2025-05-26 11:21:30.914454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00a00 of size 256 next 12 2025-05-26 11:21:30.914463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00b00 of size 256 next 13 2025-05-26 11:21:30.914472: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00c00 of size 256 next 17 2025-05-26 11:21:30.914481: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00d00 of size 256 next 19 2025-05-26 11:21:30.914490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00e00 of size 256 next 20 2025-05-26 11:21:30.914499: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce00f00 of size 256 next 21 2025-05-26 11:21:30.914507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01000 of size 256 next 22 2025-05-26 11:21:30.914516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01100 of size 256 next 24 2025-05-26 11:21:30.914525: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01200 of size 256 next 25 2025-05-26 11:21:30.914534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01300 of size 256 next 23 2025-05-26 11:21:30.914543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01400 of size 256 next 28 2025-05-26 11:21:30.914552: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01500 of size 256 next 29 2025-05-26 11:21:30.914561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01600 of size 256 next 30 2025-05-26 11:21:30.914570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01700 of size 256 next 31 2025-05-26 11:21:30.914579: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01800 of size 256 next 33 2025-05-26 11:21:30.914588: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01900 of size 256 next 34 2025-05-26 11:21:30.914597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01a00 of size 1024 next 32 2025-05-26 11:21:30.914605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce01e00 of size 1024 next 37 2025-05-26 11:21:30.914614: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce02200 of size 1024 next 38 2025-05-26 11:21:30.914623: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce02600 of size 1024 next 39 2025-05-26 11:21:30.914631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce02a00 of size 1024 next 40 2025-05-26 11:21:30.914640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce02e00 of size 1024 next 41 2025-05-26 11:21:30.914649: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce03200 of size 1024 next 43 2025-05-26 11:21:30.914658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce03600 of size 1024 next 44 2025-05-26 11:21:30.914666: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce03a00 of size 1024 next 45 2025-05-26 11:21:30.914675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce03e00 of size 1024 next 46 2025-05-26 11:21:30.914684: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04200 of size 256 next 48 2025-05-26 11:21:30.914693: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04300 of size 256 next 49 2025-05-26 11:21:30.914701: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04400 of size 256 next 50 2025-05-26 11:21:30.914710: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04500 of size 256 next 51 2025-05-26 11:21:30.914726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04600 of size 256 next 52 2025-05-26 11:21:30.914741: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04700 of size 256 next 53 2025-05-26 11:21:30.914750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04800 of size 256 next 56 2025-05-26 11:21:30.914759: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04900 of size 256 next 57 2025-05-26 11:21:30.914767: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04a00 of size 256 next 58 2025-05-26 11:21:30.914776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04b00 of size 256 next 14 2025-05-26 11:21:30.914784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04c00 of size 256 next 15 2025-05-26 11:21:30.914793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04d00 of size 256 next 16 2025-05-26 11:21:30.914801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce04e00 of size 1024 next 60 2025-05-26 11:21:30.914810: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce05200 of size 1024 next 61 2025-05-26 11:21:30.914819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce05600 of size 1024 next 62 2025-05-26 11:21:30.914827: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce05a00 of size 1024 next 63 2025-05-26 11:21:30.914836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce05e00 of size 1024 next 64 2025-05-26 11:21:30.914844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06200 of size 256 next 66 2025-05-26 11:21:30.914853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06300 of size 256 next 67 2025-05-26 11:21:30.914862: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06400 of size 256 next 68 2025-05-26 11:21:30.914870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06500 of size 256 next 69 2025-05-26 11:21:30.914879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06600 of size 256 next 70 2025-05-26 11:21:30.914887: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06700 of size 256 next 73 2025-05-26 11:21:30.914927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06800 of size 256 next 74 2025-05-26 11:21:30.914939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06900 of size 256 next 75 2025-05-26 11:21:30.914948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06a00 of size 256 next 76 2025-05-26 11:21:30.914956: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06b00 of size 256 next 77 2025-05-26 11:21:30.914964: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce06c00 of size 1024 next 79 2025-05-26 11:21:30.914972: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce07000 of size 1024 next 80 2025-05-26 11:21:30.914981: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce07400 of size 1024 next 81 2025-05-26 11:21:30.914991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce07800 of size 1024 next 82 2025-05-26 11:21:30.914999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce07c00 of size 1024 next 83 2025-05-26 11:21:30.915008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08000 of size 256 next 85 2025-05-26 11:21:30.915016: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08100 of size 256 next 86 2025-05-26 11:21:30.915026: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08200 of size 512 next 84 2025-05-26 11:21:30.915035: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08400 of size 512 next 88 2025-05-26 11:21:30.915043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08600 of size 512 next 89 2025-05-26 11:21:30.915059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08800 of size 512 next 90 2025-05-26 11:21:30.915068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08a00 of size 512 next 91 2025-05-26 11:21:30.915077: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08c00 of size 256 next 93 2025-05-26 11:21:30.915085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08d00 of size 256 next 94 2025-05-26 11:21:30.915094: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce08e00 of size 512 next 92 2025-05-26 11:21:30.915102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce09000 of size 512 next 97 2025-05-26 11:21:30.915112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce09200 of size 512 next 98 2025-05-26 11:21:30.915120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce09400 of size 512 next 99 2025-05-26 11:21:30.915129: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce09600 of size 512 next 2 2025-05-26 11:21:30.915137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce09800 of size 256 next 3 2025-05-26 11:21:30.915146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce09900 of size 256 next 4 2025-05-26 11:21:30.915155: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce09a00 of size 16384 next 18 2025-05-26 11:21:30.915164: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce0da00 of size 256 next 100 2025-05-26 11:21:30.915173: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce0db00 of size 256 next 101 2025-05-26 11:21:30.915181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce0dc00 of size 2048 next 104 2025-05-26 11:21:30.915190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce0e400 of size 2048 next 105 2025-05-26 11:21:30.915199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce0ec00 of size 2048 next 106 2025-05-26 11:21:30.915207: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce0f400 of size 2048 next 107 2025-05-26 11:21:30.915216: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce0fc00 of size 2048 next 108 2025-05-26 11:21:30.915225: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce10400 of size 256 next 109 2025-05-26 11:21:30.915233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce10500 of size 256 next 110 2025-05-26 11:21:30.915242: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce10600 of size 2048 next 113 2025-05-26 11:21:30.915250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce10e00 of size 2048 next 114 2025-05-26 11:21:30.915258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce11600 of size 2048 next 115 2025-05-26 11:21:30.915267: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce11e00 of size 2048 next 116 2025-05-26 11:21:30.915275: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce12600 of size 2048 next 6 2025-05-26 11:21:30.915284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce12e00 of size 37632 next 7 2025-05-26 11:21:30.915293: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1c100 of size 512 next 118 2025-05-26 11:21:30.915302: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1c300 of size 512 next 119 2025-05-26 11:21:30.915311: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1c500 of size 512 next 120 2025-05-26 11:21:30.915319: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1c700 of size 512 next 121 2025-05-26 11:21:30.915328: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1c900 of size 512 next 122 2025-05-26 11:21:30.915337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1cb00 of size 512 next 125 2025-05-26 11:21:30.915353: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1cd00 of size 512 next 126 2025-05-26 11:21:30.915362: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1cf00 of size 512 next 127 2025-05-26 11:21:30.915371: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1d100 of size 512 next 128 2025-05-26 11:21:30.915379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1d300 of size 512 next 129 2025-05-26 11:21:30.915388: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1d500 of size 2048 next 130 2025-05-26 11:21:30.915396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1dd00 of size 2048 next 131 2025-05-26 11:21:30.915405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1e500 of size 2048 next 132 2025-05-26 11:21:30.915414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1ed00 of size 2048 next 133 2025-05-26 11:21:30.915422: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1f500 of size 2048 next 134 2025-05-26 11:21:30.915431: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1fd00 of size 512 next 136 2025-05-26 11:21:30.915439: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce1ff00 of size 512 next 137 2025-05-26 11:21:30.915448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce20100 of size 512 next 138 2025-05-26 11:21:30.915457: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce20300 of size 512 next 139 2025-05-26 11:21:30.915465: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce20500 of size 512 next 140 2025-05-26 11:21:30.915474: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce20700 of size 512 next 141 2025-05-26 11:21:30.915483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce20900 of size 512 next 143 2025-05-26 11:21:30.915491: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce20b00 of size 512 next 144 2025-05-26 11:21:30.915500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce20d00 of size 512 next 145 2025-05-26 11:21:30.915508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce20f00 of size 512 next 146 2025-05-26 11:21:30.915517: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce21100 of size 2048 next 147 2025-05-26 11:21:30.915526: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce21900 of size 2048 next 148 2025-05-26 11:21:30.915534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce22100 of size 2048 next 149 2025-05-26 11:21:30.915543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce22900 of size 2048 next 150 2025-05-26 11:21:30.915551: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce23100 of size 2048 next 151 2025-05-26 11:21:30.915560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce23900 of size 512 next 153 2025-05-26 11:21:30.915568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce23b00 of size 512 next 154 2025-05-26 11:21:30.915577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce23d00 of size 512 next 155 2025-05-26 11:21:30.915585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce23f00 of size 512 next 156 2025-05-26 11:21:30.915594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce24100 of size 512 next 157 2025-05-26 11:21:30.915602: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce24300 of size 512 next 161 2025-05-26 11:21:30.915611: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce24500 of size 512 next 162 2025-05-26 11:21:30.915619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce24700 of size 512 next 163 2025-05-26 11:21:30.915635: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce24900 of size 512 next 164 2025-05-26 11:21:30.915644: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce24b00 of size 512 next 165 2025-05-26 11:21:30.915652: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce24d00 of size 2048 next 166 2025-05-26 11:21:30.915661: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce25500 of size 2048 next 167 2025-05-26 11:21:30.915669: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce25d00 of size 2048 next 168 2025-05-26 11:21:30.915679: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce26500 of size 2048 next 169 2025-05-26 11:21:30.915687: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce26d00 of size 2048 next 170 2025-05-26 11:21:30.915696: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce27500 of size 1024 next 172 2025-05-26 11:21:30.915705: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce27900 of size 1024 next 173 2025-05-26 11:21:30.915713: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce27d00 of size 1024 next 174 2025-05-26 11:21:30.915722: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce28100 of size 1024 next 175 2025-05-26 11:21:30.915731: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce28500 of size 1024 next 176 2025-05-26 11:21:30.915740: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce28900 of size 256 next 178 2025-05-26 11:21:30.915748: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce28a00 of size 256 next 179 2025-05-26 11:21:30.915757: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce28b00 of size 1024 next 177 2025-05-26 11:21:30.915766: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce28f00 of size 1024 next 182 2025-05-26 11:21:30.915774: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce29300 of size 1024 next 183 2025-05-26 11:21:30.915783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce29700 of size 1024 next 184 2025-05-26 11:21:30.915792: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce29b00 of size 1024 next 185 2025-05-26 11:21:30.915800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce29f00 of size 256 next 187 2025-05-26 11:21:30.915809: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce2a000 of size 256 next 188 2025-05-26 11:21:30.915818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce2a100 of size 4096 next 186 2025-05-26 11:21:30.915827: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce2b100 of size 4096 next 42 2025-05-26 11:21:30.915835: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce2c100 of size 65536 next 36 2025-05-26 11:21:30.915844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce3c100 of size 65536 next 35 2025-05-26 11:21:30.915853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce4c100 of size 4096 next 235 2025-05-26 11:21:30.915862: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce4d100 of size 4096 next 236 2025-05-26 11:21:30.915871: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce4e100 of size 4096 next 237 2025-05-26 11:21:30.915879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce4f100 of size 4096 next 238 2025-05-26 11:21:30.915888: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce50100 of size 1024 next 239 2025-05-26 11:21:30.915897: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce50500 of size 1024 next 240 2025-05-26 11:21:30.915905: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce50900 of size 1024 next 241 2025-05-26 11:21:30.915920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce50d00 of size 1024 next 242 2025-05-26 11:21:30.915930: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce51100 of size 1024 next 243 2025-05-26 11:21:30.915938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce51500 of size 1024 next 246 2025-05-26 11:21:30.915947: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce51900 of size 1024 next 247 2025-05-26 11:21:30.915955: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce51d00 of size 1024 next 248 2025-05-26 11:21:30.915964: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce52100 of size 1024 next 249 2025-05-26 11:21:30.915973: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce52500 of size 1024 next 250 2025-05-26 11:21:30.915982: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce52900 of size 4096 next 252 2025-05-26 11:21:30.915990: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce53900 of size 4096 next 253 2025-05-26 11:21:30.915999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce54900 of size 4096 next 254 2025-05-26 11:21:30.916008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce55900 of size 4096 next 255 2025-05-26 11:21:30.916016: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce56900 of size 4096 next 256 2025-05-26 11:21:30.916025: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce57900 of size 1024 next 257 2025-05-26 11:21:30.916033: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce57d00 of size 1024 next 258 2025-05-26 11:21:30.916042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce58100 of size 1024 next 259 2025-05-26 11:21:30.916050: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce58500 of size 1024 next 260 2025-05-26 11:21:30.916059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce58900 of size 1024 next 261 2025-05-26 11:21:30.916067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce58d00 of size 1024 next 264 2025-05-26 11:21:30.916076: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce59100 of size 1024 next 265 2025-05-26 11:21:30.916084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce59500 of size 1024 next 266 2025-05-26 11:21:30.916093: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce59900 of size 1024 next 267 2025-05-26 11:21:30.916101: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce59d00 of size 1024 next 268 2025-05-26 11:21:30.916110: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5a100 of size 4096 next 270 2025-05-26 11:21:30.916118: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5b100 of size 4096 next 271 2025-05-26 11:21:30.916127: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5c100 of size 4096 next 272 2025-05-26 11:21:30.916135: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5d100 of size 4096 next 273 2025-05-26 11:21:30.916143: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5e100 of size 4096 next 274 2025-05-26 11:21:30.916152: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5f100 of size 1024 next 275 2025-05-26 11:21:30.916160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5f500 of size 1024 next 276 2025-05-26 11:21:30.916169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5f900 of size 1024 next 277 2025-05-26 11:21:30.916177: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce5fd00 of size 1024 next 278 2025-05-26 11:21:30.916186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce60100 of size 1024 next 279 2025-05-26 11:21:30.916194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce60500 of size 1024 next 282 2025-05-26 11:21:30.916210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce60900 of size 1024 next 283 2025-05-26 11:21:30.916218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce60d00 of size 1024 next 284 2025-05-26 11:21:30.916227: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce61100 of size 1024 next 285 2025-05-26 11:21:30.916235: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce61500 of size 1024 next 286 2025-05-26 11:21:30.916244: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce61900 of size 4096 next 288 2025-05-26 11:21:30.916253: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce62900 of size 6144 next 27 2025-05-26 11:21:30.916262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce64100 of size 147456 next 26 2025-05-26 11:21:30.916270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce88100 of size 65536 next 47 2025-05-26 11:21:30.916279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ce98100 of size 65536 next 59 2025-05-26 11:21:30.916287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739cea8100 of size 65536 next 65 2025-05-26 11:21:30.916296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f739ceb8100 of size 294656 next 18446744073709551615 2025-05-26 11:21:30.916304: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-05-26 11:21:30.916322: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 65 Chunks of size 256 totalling 16.2KiB 2025-05-26 11:21:30.916333: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-05-26 11:21:30.916343: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 250 Chunks of size 1024 totalling 250.0KiB 2025-05-26 11:21:30.916352: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-05-26 11:21:30.916362: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 2048 totalling 80.0KiB 2025-05-26 11:21:30.916372: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 117 Chunks of size 4096 totalling 468.0KiB 2025-05-26 11:21:30.916382: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 6144 totalling 12.0KiB 2025-05-26 11:21:30.916391: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2025-05-26 11:21:30.916401: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 10 Chunks of size 8192 totalling 80.0KiB 2025-05-26 11:21:30.916410: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-05-26 11:21:30.916420: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-05-26 11:21:30.916429: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-05-26 11:21:30.916439: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-05-26 11:21:30.916448: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-05-26 11:21:30.916458: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2025-05-26 11:21:30.916467: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-05-26 11:21:30.916477: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-05-26 11:21:30.916486: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2025-05-26 11:21:30.916496: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-05-26 11:21:30.916505: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-05-26 11:21:30.916522: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1048576 totalling 25.00MiB 2025-05-26 11:21:30.916532: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 18 Chunks of size 1310720 totalling 22.50MiB 2025-05-26 11:21:30.916541: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB 2025-05-26 11:21:30.916550: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB 2025-05-26 11:21:30.916559: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 2097152 totalling 4.00MiB 2025-05-26 11:21:30.916569: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 23 Chunks of size 2359296 totalling 51.75MiB 2025-05-26 11:21:30.916578: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 4194304 totalling 8.00MiB 2025-05-26 11:21:30.916588: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 9437184 totalling 9.00MiB 2025-05-26 11:21:30.916597: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 15597568 totalling 14.88MiB 2025-05-26 11:21:30.916607: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 145.48MiB 2025-05-26 11:21:30.916616: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 185466880 memory_limit_: 185466880 available bytes: 0 curr_region_allocation_bytes_: 268435456 2025-05-26 11:21:30.916630: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 185466880 InUse: 152546816 MaxInUse: 165078528 NumAllocs: 2110 MaxAllocSize: 15597568 2025-05-26 11:21:30.916658: W tensorflow/core/common_runtime/bfc_allocator.cc:439] __*****___******___******xxx********************************_**_**************************_********* 2025-05-26 11:21:30.916717: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at random_op.cc:77 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 ERROR : 'int' object is not subscriptable reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: "'int' object is not subscriptable" #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : sam list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.19584894180297852 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:sam Mon May 26 11:21:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step sam ! Inside sam : nb paths : 1 ERROR in datou_step_exec, will save and exit ! CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2430, in datou_step_exec return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 367, in datou_step_sam sam.to(device=device) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to return self._apply(convert) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 602, in _apply param_applied = fn(param) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) [1189321094] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.012730598449707031 save_final ERROR in last step sam, CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. time spend for datou_step_exec : 5.972898960113525 time spend to save output : 0.03804373741149902 total time spend for step 0 : 6.010942697525024 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : None ERROR nb objects espect : 98 nb_objects detect : 0 ERROR sam FAILED ############################### TEST frcnn ################################ Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : frcnn list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.18572378158569336 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:frcnn Mon May 26 11:21:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step Faster rcnn ! To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] WARNING: Logging before InitGoogleLogging() is written to STDERR E0526 11:21:39.180135 3268756 common.cpp:114] Cannot create Cublas handle. Cublas won't be available. E0526 11:21:39.187915 3268756 common.cpp:121] Cannot create Curand generator. Curand won't be available. F0526 11:21:39.202283 3268756 syncedmem.hpp:22] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Aborted (core dumped) No data to report. No data to report. ret : 34304 command : coverage3 html -i --omit=/usr/local/lib/python3.8/dist-packages/*,/home/admin/.local/lib/python3.8/site-packages/*,/usr/lib/python3/dist-packages/* -d htmlcov ret : 256 command : coverage3 report -i -m ret : 256 19.50user 10.93system 1:14.45elapsed 40%CPU (0avgtext+0avgdata 2449488maxresident)k 2265672inputs+3480outputs (7675major+1243575minor)pagefaults 0swaps