python /home/admin/mtr/script_for_cron.py -j coverage -m 9 -a '' -s coverage -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded root_folder /data_4/data_log/job/2026/February/27022026/coverage/ git_velours : /home/admin/workarea/git/Velours/ out_folder_name htmlcov output_folder /data_4/data_log/job/2026/February/27022026/coverage/htmlcov new path : /data_4/data_log/job/2026/February/27022026/coverage/ command : coverage3 run /home/admin/workarea/git/Velours/python/tests/python_tests.py --short_python3 `cat ~/.fotonower_pass/bdd.py.pass` cat: /home/admin/.fotonower_pass/bdd.py.pass: Aucun fichier ou dossier de ce type import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data python version used : 3 #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 776 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.15442943572998047 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Fri Feb 27 17:20:29 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 776 wait 20 seconds l 3637 free memory gpu now : 776 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2026-02-27 17:20:52.883487: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2026-02-27 17:20:52.910424: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2026-02-27 17:20:52.911911: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe880000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2026-02-27 17:20:52.911958: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2026-02-27 17:20:52.914530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2026-02-27 17:20:53.024779: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3172fb30 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2026-02-27 17:20:53.024813: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2026-02-27 17:20:53.025419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:20:53.025708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:20:53.027783: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:20:53.029824: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:20:53.030143: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:20:53.032409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:20:53.033404: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:20:53.037939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:20:53.038877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:20:53.038955: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:20:53.039437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-02-27 17:20:53.039453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-02-27 17:20:53.039461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-02-27 17:20:53.040184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 399 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2026-02-27 17:20:53.682893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:20:53.682995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:20:53.683020: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:20:53.683051: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:20:53.683073: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:20:53.683094: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:20:53.683115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:20:53.683136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:20:53.683983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:20:53.684963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:20:53.685016: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:20:53.685041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:20:53.685066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:20:53.685097: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:20:53.685130: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:20:53.685172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:20:53.685211: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:20:53.686167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:20:53.686201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-02-27 17:20:53.686212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-02-27 17:20:53.686221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-02-27 17:20:53.687106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 399 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2026-02-27 17:20:55.332811: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.938584: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 128.00M (134217728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.939111: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 128.00M (134217728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : [] file manque in s3 : ['mask_model.h5'] 2026-02-27 17:21:12.338179: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:21:12.675918: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:21:12.760143: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2026-02-27 17:21:12.789053: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR local folder : /data/models_weight/mask_coco_origin list_images length : 1 NEW PHOTO Processing 1 images image shape: (480, 640, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 640.00000 error in detect the image : temp/1772209229_1427270_957285035_a42482e51c93c8025d243dd179aee85b.jpg 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node conv1/convolution (defined at usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:3007) ]] [[ROI/strided_slice_19/_20]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node conv1/convolution (defined at usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:3007) ]] 0 successful operations. 0 derived errors ignored. [Op:__inference_keras_scratch_graph_13584] Function call stack: keras_scratch_graph -> keras_scratch_graph Detection mask done ! Trying to reset tf kernel 1427497 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 17 tf kernel not reseted sub process len(results) : 0 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 776 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] WARNING : results is empty ! time spent for convertir_results : 0.7311558723449707 time spend for datou_step_exec : 47.03843426704407 time spend to save output : 3.9577484130859375e-05 total time spend for step 1 : 47.0384738445282 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.018929243087768555 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'957285035': [[], 'temp/1772209229_1427270_957285035_a42482e51c93c8025d243dd179aee85b.jpg']} free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 776 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.1623067855834961 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Fri Feb 27 17:21:17 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 776 wait 20 seconds l 3637 free memory gpu now : 776 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2026-02-27 17:21:41.228825: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2026-02-27 17:21:41.254499: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2026-02-27 17:21:41.256829: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe884000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2026-02-27 17:21:41.256892: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2026-02-27 17:21:41.261062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2026-02-27 17:21:41.377015: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x31ab6590 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2026-02-27 17:21:41.377060: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2026-02-27 17:21:41.377961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:21:41.378399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:21:41.381810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:21:41.384874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:21:41.385405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:21:41.388476: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:21:41.389482: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:21:41.394012: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:21:41.394914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:21:41.394995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:21:41.395458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-02-27 17:21:41.395473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-02-27 17:21:41.395481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-02-27 17:21:41.396211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 399 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2026-02-27 17:21:41.503404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:21:41.503551: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:21:41.503577: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:21:41.503600: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:21:41.503623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:21:41.503645: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:21:41.503667: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:21:41.503689: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:21:41.504561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:21:41.505629: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:21:41.505673: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:21:41.505699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:21:41.505723: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:21:41.505748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:21:41.505772: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:21:41.505796: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:21:41.505820: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:21:41.506713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:21:41.506755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-02-27 17:21:41.506767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-02-27 17:21:41.506777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-02-27 17:21:41.507684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 399 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead 2026-02-27 17:21:54.063233: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 49.00MiB (rounded to 51380224) Current allocation summary follows. 2026-02-27 17:21:54.063342: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2026-02-27 17:21:54.063363: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 77, Chunks in use: 77. 19.2KiB allocated for chunks. 19.2KiB in use in bin. 9.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063398: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063415: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 259, Chunks in use: 259. 259.2KiB allocated for chunks. 259.2KiB in use in bin. 259.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063431: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 56, Chunks in use: 56. 112.0KiB allocated for chunks. 112.0KiB in use in bin. 112.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063447: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 120, Chunks in use: 120. 487.5KiB allocated for chunks. 487.5KiB in use in bin. 480.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063463: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 23, Chunks in use: 21. 194.8KiB allocated for chunks. 172.0KiB in use in bin. 172.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063479: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 2, Chunks in use: 2. 40.0KiB allocated for chunks. 40.0KiB in use in bin. 40.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063494: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 2, Chunks in use: 1. 84.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2026-02-27 17:21:54.063510: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063526: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 5, Chunks in use: 4. 783.2KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2026-02-27 17:21:54.063541: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 9, Chunks in use: 8. 2.66MiB allocated for chunks. 2.16MiB in use in bin. 2.00MiB client-requested in use in bin. 2026-02-27 17:21:54.063556: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 8, Chunks in use: 7. 4.64MiB allocated for chunks. 3.94MiB in use in bin. 3.75MiB client-requested in use in bin. 2026-02-27 17:21:54.063572: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 47, Chunks in use: 46. 53.12MiB allocated for chunks. 51.88MiB in use in bin. 46.00MiB client-requested in use in bin. 2026-02-27 17:21:54.063587: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 31, Chunks in use: 29. 71.00MiB allocated for chunks. 66.25MiB in use in bin. 64.50MiB client-requested in use in bin. 2026-02-27 17:21:54.063603: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 10, Chunks in use: 7. 46.25MiB allocated for chunks. 30.50MiB in use in bin. 26.75MiB client-requested in use in bin. 2026-02-27 17:21:54.063618: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 4, Chunks in use: 4. 35.00MiB allocated for chunks. 35.00MiB in use in bin. 35.00MiB client-requested in use in bin. 2026-02-27 17:21:54.063633: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:54.063648: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 2, Chunks in use: 1. 89.00MiB allocated for chunks. 49.00MiB in use in bin. 49.00MiB client-requested in use in bin. 2026-02-27 17:21:54.063664: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 1, Chunks in use: 1. 95.69MiB allocated for chunks. 95.69MiB in use in bin. 49.00MiB client-requested in use in bin. 2026-02-27 17:21:54.063687: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:54.063701: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:54.063716: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 49.00MiB was 32.00MiB, Chunk State: 2026-02-27 17:21:54.063738: I tensorflow/core/common_runtime/bfc_allocator.cc:963] Size: 40.00MiB | Requested Size: 0B | in_use: 0 | bin_num: 17, prev: Size: 4.50MiB | Requested Size: 4.50MiB | in_use: 1 | bin_num: -1 2026-02-27 17:21:54.063750: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 151715840 2026-02-27 17:21:54.063768: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe696000000 of size 51380224 next 699 2026-02-27 17:21:54.063781: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe699100000 of size 100335616 next 18446744073709551615 2026-02-27 17:21:54.063793: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 134217728 2026-02-27 17:21:54.063806: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6a0000000 of size 4194304 next 632 2026-02-27 17:21:54.063818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a0400000 of size 4194304 next 617 2026-02-27 17:21:54.063831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a0800000 of size 4194304 next 616 2026-02-27 17:21:54.063843: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a0c00000 of size 6291456 next 608 2026-02-27 17:21:54.063856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a1200000 of size 9437184 next 607 2026-02-27 17:21:54.063869: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a1b00000 of size 4194304 next 644 2026-02-27 17:21:54.063881: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a1f00000 of size 4194304 next 625 2026-02-27 17:21:54.063893: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a2300000 of size 8388608 next 624 2026-02-27 17:21:54.063906: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a2b00000 of size 4194304 next 661 2026-02-27 17:21:54.063918: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a2f00000 of size 2359296 next 676 2026-02-27 17:21:54.063930: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a3140000 of size 2359296 next 678 2026-02-27 17:21:54.063942: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a3380000 of size 2359296 next 680 2026-02-27 17:21:54.063955: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6a35c0000 of size 7602176 next 639 2026-02-27 17:21:54.063967: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a3d00000 of size 9437184 next 638 2026-02-27 17:21:54.063979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a4600000 of size 9437184 next 656 2026-02-27 17:21:54.063991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6a4f00000 of size 4718592 next 685 2026-02-27 17:21:54.064004: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a5380000 of size 4718592 next 684 2026-02-27 17:21:54.064016: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6a5800000 of size 41943040 next 18446744073709551615 2026-02-27 17:21:54.064028: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 67108864 2026-02-27 17:21:54.064041: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a8000000 of size 2359296 next 370 2026-02-27 17:21:54.064053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a8240000 of size 1048576 next 394 2026-02-27 17:21:54.064074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6a8340000 of size 1310720 next 388 2026-02-27 17:21:54.064087: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a8480000 of size 2359296 next 387 2026-02-27 17:21:54.064099: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a86c0000 of size 1048576 next 412 2026-02-27 17:21:54.064112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a87c0000 of size 1310720 next 406 2026-02-27 17:21:54.064124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a8900000 of size 2359296 next 405 2026-02-27 17:21:54.064136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a8b40000 of size 1048576 next 430 2026-02-27 17:21:54.064149: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a8c40000 of size 1310720 next 424 2026-02-27 17:21:54.064161: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a8d80000 of size 2359296 next 423 2026-02-27 17:21:54.064173: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a8fc0000 of size 1048576 next 447 2026-02-27 17:21:54.064185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a90c0000 of size 1310720 next 441 2026-02-27 17:21:54.064198: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a9200000 of size 2359296 next 440 2026-02-27 17:21:54.064210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a9440000 of size 1048576 next 465 2026-02-27 17:21:54.064222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a9540000 of size 1310720 next 459 2026-02-27 17:21:54.064234: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a9680000 of size 2359296 next 458 2026-02-27 17:21:54.064246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a98c0000 of size 1048576 next 483 2026-02-27 17:21:54.064259: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a99c0000 of size 1310720 next 477 2026-02-27 17:21:54.064271: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a9b00000 of size 2359296 next 476 2026-02-27 17:21:54.064283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a9d40000 of size 1048576 next 501 2026-02-27 17:21:54.064295: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a9e40000 of size 1310720 next 495 2026-02-27 17:21:54.064307: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6a9f80000 of size 2359296 next 494 2026-02-27 17:21:54.064320: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aa1c0000 of size 1048576 next 519 2026-02-27 17:21:54.064332: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aa2c0000 of size 1310720 next 513 2026-02-27 17:21:54.064344: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aa400000 of size 2359296 next 512 2026-02-27 17:21:54.064356: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aa640000 of size 1048576 next 537 2026-02-27 17:21:54.064368: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aa740000 of size 1310720 next 531 2026-02-27 17:21:54.064381: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aa880000 of size 2359296 next 530 2026-02-27 17:21:54.064393: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aaac0000 of size 1048576 next 555 2026-02-27 17:21:54.064405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aabc0000 of size 1310720 next 549 2026-02-27 17:21:54.064417: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aad00000 of size 2359296 next 548 2026-02-27 17:21:54.064429: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aaf40000 of size 1048576 next 573 2026-02-27 17:21:54.064442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ab040000 of size 1310720 next 567 2026-02-27 17:21:54.064462: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ab180000 of size 2359296 next 566 2026-02-27 17:21:54.064475: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ab3c0000 of size 1048576 next 591 2026-02-27 17:21:54.064487: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ab4c0000 of size 1310720 next 585 2026-02-27 17:21:54.064500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ab600000 of size 2359296 next 584 2026-02-27 17:21:54.064512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6ab840000 of size 2097152 next 597 2026-02-27 17:21:54.064524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aba40000 of size 2097152 next 596 2026-02-27 17:21:54.064537: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6abc40000 of size 3932160 next 18446744073709551615 2026-02-27 17:21:54.064549: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432 2026-02-27 17:21:54.064561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ac000000 of size 2359296 next 245 2026-02-27 17:21:54.064573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ac240000 of size 1048576 next 269 2026-02-27 17:21:54.064586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ac340000 of size 1310720 next 263 2026-02-27 17:21:54.064598: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ac480000 of size 2359296 next 262 2026-02-27 17:21:54.064610: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ac6c0000 of size 1048576 next 287 2026-02-27 17:21:54.064622: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ac7c0000 of size 1310720 next 281 2026-02-27 17:21:54.064634: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ac900000 of size 2359296 next 280 2026-02-27 17:21:54.064647: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6acb40000 of size 1048576 next 304 2026-02-27 17:21:54.064659: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6acc40000 of size 1310720 next 298 2026-02-27 17:21:54.064671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6acd80000 of size 2359296 next 297 2026-02-27 17:21:54.064683: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6acfc0000 of size 1048576 next 322 2026-02-27 17:21:54.064695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ad0c0000 of size 1310720 next 316 2026-02-27 17:21:54.064708: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ad200000 of size 2359296 next 315 2026-02-27 17:21:54.064720: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ad440000 of size 1048576 next 340 2026-02-27 17:21:54.064732: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ad540000 of size 1310720 next 334 2026-02-27 17:21:54.064745: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ad680000 of size 2359296 next 333 2026-02-27 17:21:54.064757: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ad8c0000 of size 1048576 next 358 2026-02-27 17:21:54.064769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ad9c0000 of size 1310720 next 352 2026-02-27 17:21:54.064781: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6adb00000 of size 2359296 next 351 2026-02-27 17:21:54.064793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6add40000 of size 1048576 next 376 2026-02-27 17:21:54.064806: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ade40000 of size 1835008 next 18446744073709551615 2026-02-27 17:21:54.064818: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216 2026-02-27 17:21:54.064830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ae000000 of size 2359296 next 181 2026-02-27 17:21:54.064842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ae240000 of size 2097152 next 196 2026-02-27 17:21:54.064863: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ae440000 of size 1048576 next 216 2026-02-27 17:21:54.064876: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ae540000 of size 1310720 next 209 2026-02-27 17:21:54.064888: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ae680000 of size 2359296 next 208 2026-02-27 17:21:54.064900: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ae8c0000 of size 1048576 next 234 2026-02-27 17:21:54.064913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6ae9c0000 of size 1310720 next 228 2026-02-27 17:21:54.064925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6aeb00000 of size 2359296 next 227 2026-02-27 17:21:54.064937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6aed40000 of size 2883584 next 18446744073709551615 2026-02-27 17:21:54.064949: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2026-02-27 17:21:54.064961: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0600000 of size 147456 next 55 2026-02-27 17:21:54.064974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0624000 of size 65536 next 78 2026-02-27 17:21:54.064986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0634000 of size 4096 next 191 2026-02-27 17:21:54.064999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0635000 of size 4096 next 192 2026-02-27 17:21:54.065011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0636000 of size 4096 next 193 2026-02-27 17:21:54.065023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0637000 of size 256 next 194 2026-02-27 17:21:54.065035: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0637100 of size 256 next 195 2026-02-27 17:21:54.065048: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0637200 of size 4096 next 197 2026-02-27 17:21:54.065060: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0638200 of size 4096 next 198 2026-02-27 17:21:54.065072: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0639200 of size 4096 next 199 2026-02-27 17:21:54.065084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063a200 of size 4096 next 200 2026-02-27 17:21:54.065096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063b200 of size 4096 next 201 2026-02-27 17:21:54.065109: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063c200 of size 1024 next 202 2026-02-27 17:21:54.065121: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063c600 of size 1024 next 204 2026-02-27 17:21:54.065133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063ca00 of size 1024 next 205 2026-02-27 17:21:54.065145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063ce00 of size 1024 next 206 2026-02-27 17:21:54.065158: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063d200 of size 1024 next 207 2026-02-27 17:21:54.065170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063d600 of size 1024 next 210 2026-02-27 17:21:54.065182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063da00 of size 1024 next 211 2026-02-27 17:21:54.065194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063de00 of size 1024 next 212 2026-02-27 17:21:54.065206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063e200 of size 1024 next 213 2026-02-27 17:21:54.065218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063e600 of size 1024 next 214 2026-02-27 17:21:54.065231: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063ea00 of size 4096 next 215 2026-02-27 17:21:54.065243: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b063fa00 of size 4096 next 217 2026-02-27 17:21:54.065263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0640a00 of size 4096 next 218 2026-02-27 17:21:54.065276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0641a00 of size 4096 next 219 2026-02-27 17:21:54.065288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0642a00 of size 4096 next 220 2026-02-27 17:21:54.065300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0643a00 of size 1024 next 221 2026-02-27 17:21:54.065313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0643e00 of size 1024 next 223 2026-02-27 17:21:54.065325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0644200 of size 1024 next 224 2026-02-27 17:21:54.065337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0644600 of size 1024 next 225 2026-02-27 17:21:54.065349: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0644a00 of size 1024 next 226 2026-02-27 17:21:54.065361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0644e00 of size 1024 next 229 2026-02-27 17:21:54.065374: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0645200 of size 1024 next 230 2026-02-27 17:21:54.065386: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0645600 of size 1024 next 231 2026-02-27 17:21:54.065398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0645a00 of size 1024 next 232 2026-02-27 17:21:54.065410: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0645e00 of size 1024 next 233 2026-02-27 17:21:54.065423: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0646200 of size 7680 next 72 2026-02-27 17:21:54.065435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0648000 of size 147456 next 71 2026-02-27 17:21:54.065447: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b066c000 of size 131072 next 87 2026-02-27 17:21:54.065459: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b068c000 of size 4096 next 289 2026-02-27 17:21:54.065472: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b068d000 of size 4096 next 290 2026-02-27 17:21:54.065484: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b068e000 of size 4096 next 291 2026-02-27 17:21:54.065496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b068f000 of size 1024 next 292 2026-02-27 17:21:54.065508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b068f400 of size 1024 next 293 2026-02-27 17:21:54.065521: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b068f800 of size 1024 next 294 2026-02-27 17:21:54.065533: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b068fc00 of size 1024 next 295 2026-02-27 17:21:54.065545: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0690000 of size 1024 next 296 2026-02-27 17:21:54.065557: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0690400 of size 1024 next 299 2026-02-27 17:21:54.065569: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0690800 of size 1024 next 300 2026-02-27 17:21:54.065581: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0690c00 of size 1024 next 301 2026-02-27 17:21:54.065594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0691000 of size 1024 next 302 2026-02-27 17:21:54.065606: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0691400 of size 1024 next 303 2026-02-27 17:21:54.065618: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0691800 of size 4096 next 305 2026-02-27 17:21:54.065630: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0692800 of size 4096 next 306 2026-02-27 17:21:54.065642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0693800 of size 4096 next 307 2026-02-27 17:21:54.065663: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0694800 of size 4096 next 308 2026-02-27 17:21:54.065676: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0695800 of size 4096 next 309 2026-02-27 17:21:54.065688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0696800 of size 1024 next 310 2026-02-27 17:21:54.065700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0696c00 of size 1024 next 311 2026-02-27 17:21:54.065712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0697000 of size 1024 next 312 2026-02-27 17:21:54.065725: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0697400 of size 1024 next 313 2026-02-27 17:21:54.065737: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0697800 of size 1024 next 314 2026-02-27 17:21:54.065749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0697c00 of size 1024 next 317 2026-02-27 17:21:54.065761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0698000 of size 1024 next 318 2026-02-27 17:21:54.065773: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0698400 of size 1024 next 319 2026-02-27 17:21:54.065785: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0698800 of size 1024 next 320 2026-02-27 17:21:54.065798: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0698c00 of size 1024 next 321 2026-02-27 17:21:54.065810: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0699000 of size 4096 next 323 2026-02-27 17:21:54.065822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069a000 of size 4096 next 324 2026-02-27 17:21:54.065834: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069b000 of size 4096 next 325 2026-02-27 17:21:54.065846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069c000 of size 4096 next 326 2026-02-27 17:21:54.065858: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069d000 of size 4096 next 327 2026-02-27 17:21:54.065871: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069e000 of size 1024 next 328 2026-02-27 17:21:54.065883: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069e400 of size 1024 next 329 2026-02-27 17:21:54.065895: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069e800 of size 1024 next 330 2026-02-27 17:21:54.065907: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069ec00 of size 1024 next 331 2026-02-27 17:21:54.065919: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069f000 of size 1024 next 332 2026-02-27 17:21:54.065932: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069f400 of size 1024 next 335 2026-02-27 17:21:54.065944: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069f800 of size 1024 next 336 2026-02-27 17:21:54.065956: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b069fc00 of size 1024 next 337 2026-02-27 17:21:54.065968: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a0000 of size 1024 next 338 2026-02-27 17:21:54.065980: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a0400 of size 1024 next 339 2026-02-27 17:21:54.065993: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a0800 of size 4096 next 341 2026-02-27 17:21:54.066005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a1800 of size 4096 next 342 2026-02-27 17:21:54.066017: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a2800 of size 4096 next 343 2026-02-27 17:21:54.066029: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a3800 of size 4096 next 344 2026-02-27 17:21:54.066041: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a4800 of size 4096 next 345 2026-02-27 17:21:54.066053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a5800 of size 1024 next 346 2026-02-27 17:21:54.066073: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a5c00 of size 1024 next 347 2026-02-27 17:21:54.066086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a6000 of size 1024 next 348 2026-02-27 17:21:54.066098: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a6400 of size 1024 next 349 2026-02-27 17:21:54.066111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a6800 of size 1024 next 350 2026-02-27 17:21:54.066123: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a6c00 of size 1024 next 353 2026-02-27 17:21:54.066135: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a7000 of size 1024 next 354 2026-02-27 17:21:54.066147: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a7400 of size 1024 next 355 2026-02-27 17:21:54.066159: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a7800 of size 1024 next 356 2026-02-27 17:21:54.066172: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a7c00 of size 1024 next 357 2026-02-27 17:21:54.066184: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a8000 of size 4096 next 359 2026-02-27 17:21:54.066196: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06a9000 of size 4096 next 360 2026-02-27 17:21:54.066208: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06aa000 of size 4096 next 361 2026-02-27 17:21:54.066221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ab000 of size 4096 next 362 2026-02-27 17:21:54.066233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ac000 of size 4096 next 363 2026-02-27 17:21:54.066245: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ad000 of size 1024 next 364 2026-02-27 17:21:54.066257: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ad400 of size 1024 next 365 2026-02-27 17:21:54.066269: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ad800 of size 1024 next 366 2026-02-27 17:21:54.066300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06adc00 of size 1024 next 367 2026-02-27 17:21:54.066312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ae000 of size 1024 next 368 2026-02-27 17:21:54.066325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ae400 of size 1024 next 371 2026-02-27 17:21:54.066337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ae800 of size 1024 next 372 2026-02-27 17:21:54.066349: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06aec00 of size 1024 next 373 2026-02-27 17:21:54.066361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06af000 of size 1024 next 374 2026-02-27 17:21:54.066373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06af400 of size 1024 next 375 2026-02-27 17:21:54.066386: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06af800 of size 4096 next 377 2026-02-27 17:21:54.066398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b0800 of size 4096 next 378 2026-02-27 17:21:54.066410: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b1800 of size 4096 next 379 2026-02-27 17:21:54.066422: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b2800 of size 4096 next 380 2026-02-27 17:21:54.066434: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b3800 of size 4096 next 381 2026-02-27 17:21:54.066446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b4800 of size 1024 next 382 2026-02-27 17:21:54.066459: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b4c00 of size 1024 next 383 2026-02-27 17:21:54.066471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b5000 of size 1024 next 384 2026-02-27 17:21:54.066492: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b5400 of size 1024 next 385 2026-02-27 17:21:54.066505: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b5800 of size 1024 next 386 2026-02-27 17:21:54.066517: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b5c00 of size 1024 next 389 2026-02-27 17:21:54.066529: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b6000 of size 1024 next 390 2026-02-27 17:21:54.066541: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b6400 of size 1024 next 391 2026-02-27 17:21:54.066553: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b6800 of size 1024 next 392 2026-02-27 17:21:54.066566: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b6c00 of size 1024 next 393 2026-02-27 17:21:54.066578: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b7000 of size 4096 next 395 2026-02-27 17:21:54.066590: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b8000 of size 4096 next 396 2026-02-27 17:21:54.066602: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06b9000 of size 4096 next 397 2026-02-27 17:21:54.066614: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ba000 of size 4096 next 398 2026-02-27 17:21:54.066627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bb000 of size 4096 next 399 2026-02-27 17:21:54.066639: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bc000 of size 1024 next 400 2026-02-27 17:21:54.066651: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bc400 of size 1024 next 401 2026-02-27 17:21:54.066663: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bc800 of size 1024 next 402 2026-02-27 17:21:54.066675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bcc00 of size 1024 next 403 2026-02-27 17:21:54.066688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bd000 of size 1024 next 404 2026-02-27 17:21:54.066700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bd400 of size 1024 next 407 2026-02-27 17:21:54.066712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bd800 of size 1024 next 408 2026-02-27 17:21:54.066724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bdc00 of size 1024 next 409 2026-02-27 17:21:54.066736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06be000 of size 1024 next 410 2026-02-27 17:21:54.066749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06be400 of size 1024 next 411 2026-02-27 17:21:54.066761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06be800 of size 4096 next 413 2026-02-27 17:21:54.066773: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06bf800 of size 4096 next 414 2026-02-27 17:21:54.066785: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c0800 of size 4096 next 415 2026-02-27 17:21:54.066797: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c1800 of size 4096 next 416 2026-02-27 17:21:54.066810: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c2800 of size 4096 next 417 2026-02-27 17:21:54.066822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c3800 of size 1024 next 418 2026-02-27 17:21:54.066834: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c3c00 of size 1024 next 419 2026-02-27 17:21:54.066846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c4000 of size 1024 next 420 2026-02-27 17:21:54.066858: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c4400 of size 1024 next 421 2026-02-27 17:21:54.066871: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c4800 of size 1024 next 422 2026-02-27 17:21:54.066891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c4c00 of size 1024 next 425 2026-02-27 17:21:54.066903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c5000 of size 1024 next 426 2026-02-27 17:21:54.066915: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c5400 of size 1024 next 427 2026-02-27 17:21:54.066928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c5800 of size 1024 next 428 2026-02-27 17:21:54.066940: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c5c00 of size 1024 next 429 2026-02-27 17:21:54.066952: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c6000 of size 4096 next 431 2026-02-27 17:21:54.066964: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c7000 of size 4096 next 432 2026-02-27 17:21:54.066976: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c8000 of size 4096 next 433 2026-02-27 17:21:54.066989: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06c9000 of size 4096 next 434 2026-02-27 17:21:54.067001: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06ca000 of size 4096 next 435 2026-02-27 17:21:54.067013: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06cb000 of size 1024 next 436 2026-02-27 17:21:54.067025: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06cb400 of size 1024 next 437 2026-02-27 17:21:54.067037: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06cb800 of size 1024 next 438 2026-02-27 17:21:54.067050: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06cbc00 of size 1024 next 103 2026-02-27 17:21:54.067062: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b06cc000 of size 262144 next 102 2026-02-27 17:21:54.067074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b070c000 of size 262144 next 674 2026-02-27 17:21:54.067087: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6b074c000 of size 737280 next 18446744073709551615 2026-02-27 17:21:54.067099: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2026-02-27 17:21:54.067111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0800000 of size 589824 next 96 2026-02-27 17:21:54.067124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0890000 of size 262144 next 117 2026-02-27 17:21:54.067136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d0000 of size 1024 next 439 2026-02-27 17:21:54.067148: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d0400 of size 1024 next 442 2026-02-27 17:21:54.067160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d0800 of size 1024 next 443 2026-02-27 17:21:54.067172: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d0c00 of size 1024 next 444 2026-02-27 17:21:54.067185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d1000 of size 1024 next 445 2026-02-27 17:21:54.067197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d1400 of size 1024 next 446 2026-02-27 17:21:54.067209: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d1800 of size 4096 next 448 2026-02-27 17:21:54.067221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d2800 of size 4096 next 449 2026-02-27 17:21:54.067233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d3800 of size 4096 next 450 2026-02-27 17:21:54.067246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d4800 of size 4096 next 451 2026-02-27 17:21:54.067258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d5800 of size 4096 next 452 2026-02-27 17:21:54.067270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d6800 of size 1024 next 453 2026-02-27 17:21:54.067290: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d6c00 of size 1024 next 454 2026-02-27 17:21:54.067303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d7000 of size 1024 next 455 2026-02-27 17:21:54.067315: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d7400 of size 1024 next 456 2026-02-27 17:21:54.067328: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d7800 of size 1024 next 457 2026-02-27 17:21:54.067340: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d7c00 of size 1024 next 460 2026-02-27 17:21:54.067352: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d8000 of size 1024 next 461 2026-02-27 17:21:54.067364: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d8400 of size 1024 next 462 2026-02-27 17:21:54.067377: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d8800 of size 1024 next 463 2026-02-27 17:21:54.067389: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d8c00 of size 1024 next 464 2026-02-27 17:21:54.067401: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08d9000 of size 4096 next 466 2026-02-27 17:21:54.067413: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08da000 of size 4096 next 467 2026-02-27 17:21:54.067425: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08db000 of size 4096 next 468 2026-02-27 17:21:54.067438: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08dc000 of size 4096 next 469 2026-02-27 17:21:54.067450: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08dd000 of size 4096 next 470 2026-02-27 17:21:54.067462: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08de000 of size 1024 next 471 2026-02-27 17:21:54.067474: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08de400 of size 1024 next 472 2026-02-27 17:21:54.067486: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08de800 of size 1024 next 473 2026-02-27 17:21:54.067498: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08dec00 of size 1024 next 474 2026-02-27 17:21:54.067511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08df000 of size 1024 next 475 2026-02-27 17:21:54.067523: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08df400 of size 1024 next 478 2026-02-27 17:21:54.067535: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08df800 of size 1024 next 479 2026-02-27 17:21:54.067547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08dfc00 of size 1024 next 480 2026-02-27 17:21:54.067559: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e0000 of size 1024 next 481 2026-02-27 17:21:54.067572: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e0400 of size 1024 next 482 2026-02-27 17:21:54.067584: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e0800 of size 4096 next 484 2026-02-27 17:21:54.067596: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e1800 of size 4096 next 485 2026-02-27 17:21:54.067608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e2800 of size 4096 next 486 2026-02-27 17:21:54.067620: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e3800 of size 4096 next 487 2026-02-27 17:21:54.067632: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e4800 of size 4096 next 488 2026-02-27 17:21:54.067645: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e5800 of size 1024 next 489 2026-02-27 17:21:54.067657: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e5c00 of size 1024 next 490 2026-02-27 17:21:54.067669: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e6000 of size 1024 next 491 2026-02-27 17:21:54.067689: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e6400 of size 1024 next 492 2026-02-27 17:21:54.067702: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e6800 of size 1024 next 493 2026-02-27 17:21:54.067714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e6c00 of size 1024 next 496 2026-02-27 17:21:54.067727: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e7000 of size 1024 next 497 2026-02-27 17:21:54.067739: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e7400 of size 1024 next 498 2026-02-27 17:21:54.067751: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e7800 of size 1024 next 499 2026-02-27 17:21:54.067763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e7c00 of size 1024 next 500 2026-02-27 17:21:54.067776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e8000 of size 4096 next 502 2026-02-27 17:21:54.067788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08e9000 of size 4096 next 503 2026-02-27 17:21:54.067800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ea000 of size 4096 next 504 2026-02-27 17:21:54.067812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08eb000 of size 4096 next 505 2026-02-27 17:21:54.067824: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ec000 of size 4096 next 506 2026-02-27 17:21:54.067837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ed000 of size 1024 next 507 2026-02-27 17:21:54.067849: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ed400 of size 1024 next 508 2026-02-27 17:21:54.067861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ed800 of size 1024 next 509 2026-02-27 17:21:54.067873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08edc00 of size 1024 next 510 2026-02-27 17:21:54.067885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ee000 of size 1024 next 511 2026-02-27 17:21:54.067898: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ee400 of size 1024 next 514 2026-02-27 17:21:54.067910: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ee800 of size 1024 next 515 2026-02-27 17:21:54.067922: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08eec00 of size 1024 next 516 2026-02-27 17:21:54.067934: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ef000 of size 1024 next 517 2026-02-27 17:21:54.067946: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ef400 of size 1024 next 518 2026-02-27 17:21:54.067959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ef800 of size 4096 next 520 2026-02-27 17:21:54.067971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f0800 of size 4096 next 521 2026-02-27 17:21:54.067983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f1800 of size 4096 next 522 2026-02-27 17:21:54.067995: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f2800 of size 4096 next 523 2026-02-27 17:21:54.068007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f3800 of size 4096 next 524 2026-02-27 17:21:54.068020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f4800 of size 1024 next 525 2026-02-27 17:21:54.068032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f4c00 of size 1024 next 526 2026-02-27 17:21:54.068044: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f5000 of size 1024 next 527 2026-02-27 17:21:54.068056: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f5400 of size 1024 next 528 2026-02-27 17:21:54.068068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f5800 of size 1024 next 529 2026-02-27 17:21:54.068081: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f5c00 of size 1024 next 532 2026-02-27 17:21:54.068101: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f6000 of size 1024 next 533 2026-02-27 17:21:54.068113: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f6400 of size 1024 next 534 2026-02-27 17:21:54.068125: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f6800 of size 1024 next 535 2026-02-27 17:21:54.068138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f6c00 of size 1024 next 536 2026-02-27 17:21:54.068150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f7000 of size 4096 next 538 2026-02-27 17:21:54.068162: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f8000 of size 4096 next 539 2026-02-27 17:21:54.068174: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08f9000 of size 4096 next 540 2026-02-27 17:21:54.068187: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fa000 of size 4096 next 541 2026-02-27 17:21:54.068199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fb000 of size 4096 next 542 2026-02-27 17:21:54.068211: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fc000 of size 1024 next 543 2026-02-27 17:21:54.068223: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fc400 of size 1024 next 544 2026-02-27 17:21:54.068235: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fc800 of size 1024 next 545 2026-02-27 17:21:54.068248: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fcc00 of size 1024 next 546 2026-02-27 17:21:54.068260: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fd000 of size 1024 next 547 2026-02-27 17:21:54.068272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fd400 of size 1024 next 550 2026-02-27 17:21:54.068284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fd800 of size 1024 next 551 2026-02-27 17:21:54.068296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fdc00 of size 1024 next 552 2026-02-27 17:21:54.068309: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fe000 of size 1024 next 553 2026-02-27 17:21:54.068321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fe400 of size 1024 next 554 2026-02-27 17:21:54.068333: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08fe800 of size 4096 next 556 2026-02-27 17:21:54.068345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b08ff800 of size 4096 next 557 2026-02-27 17:21:54.068357: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0900800 of size 4096 next 558 2026-02-27 17:21:54.068369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0901800 of size 4096 next 559 2026-02-27 17:21:54.068382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0902800 of size 4096 next 560 2026-02-27 17:21:54.068394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0903800 of size 1024 next 561 2026-02-27 17:21:54.068406: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0903c00 of size 1024 next 562 2026-02-27 17:21:54.068418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0904000 of size 1024 next 563 2026-02-27 17:21:54.068430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0904400 of size 1024 next 564 2026-02-27 17:21:54.068443: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0904800 of size 1024 next 565 2026-02-27 17:21:54.068455: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0904c00 of size 1024 next 568 2026-02-27 17:21:54.068467: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0905000 of size 1024 next 569 2026-02-27 17:21:54.068479: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0905400 of size 1024 next 570 2026-02-27 17:21:54.068499: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0905800 of size 1024 next 571 2026-02-27 17:21:54.068512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0905c00 of size 1024 next 572 2026-02-27 17:21:54.068524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0906000 of size 4096 next 574 2026-02-27 17:21:54.068536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0907000 of size 4096 next 575 2026-02-27 17:21:54.068548: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0908000 of size 4096 next 576 2026-02-27 17:21:54.068561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0909000 of size 4096 next 577 2026-02-27 17:21:54.068573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090a000 of size 4096 next 578 2026-02-27 17:21:54.068585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090b000 of size 1024 next 579 2026-02-27 17:21:54.068597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090b400 of size 1024 next 580 2026-02-27 17:21:54.068609: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090b800 of size 1024 next 581 2026-02-27 17:21:54.068621: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090bc00 of size 1024 next 582 2026-02-27 17:21:54.068634: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090c000 of size 1024 next 583 2026-02-27 17:21:54.068646: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090c400 of size 1024 next 586 2026-02-27 17:21:54.068658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090c800 of size 1024 next 587 2026-02-27 17:21:54.068670: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090cc00 of size 1024 next 588 2026-02-27 17:21:54.068682: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090d000 of size 1024 next 589 2026-02-27 17:21:54.068695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090d400 of size 1024 next 590 2026-02-27 17:21:54.068707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090d800 of size 4096 next 592 2026-02-27 17:21:54.068719: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b090e800 of size 6144 next 112 2026-02-27 17:21:54.068732: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0910000 of size 524288 next 111 2026-02-27 17:21:54.068744: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0990000 of size 262144 next 135 2026-02-27 17:21:54.068757: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b09d0000 of size 327680 next 124 2026-02-27 17:21:54.068769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0a20000 of size 589824 next 123 2026-02-27 17:21:54.068781: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0ab0000 of size 262144 next 152 2026-02-27 17:21:54.068793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0af0000 of size 327680 next 142 2026-02-27 17:21:54.068806: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0b40000 of size 786432 next 18446744073709551615 2026-02-27 17:21:54.068818: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2026-02-27 17:21:54.068830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c00000 of size 4096 next 593 2026-02-27 17:21:54.068842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c01000 of size 4096 next 594 2026-02-27 17:21:54.068855: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c02000 of size 4096 next 595 2026-02-27 17:21:54.068867: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c03000 of size 2048 next 598 2026-02-27 17:21:54.068879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c03800 of size 2048 next 599 2026-02-27 17:21:54.068903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c04000 of size 2048 next 600 2026-02-27 17:21:54.068916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c04800 of size 2048 next 601 2026-02-27 17:21:54.068928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c05000 of size 2048 next 602 2026-02-27 17:21:54.068941: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c05800 of size 256 next 605 2026-02-27 17:21:54.068953: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c05900 of size 256 next 606 2026-02-27 17:21:54.068965: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c05a00 of size 2048 next 604 2026-02-27 17:21:54.068977: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c06200 of size 2048 next 609 2026-02-27 17:21:54.068989: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c06a00 of size 2048 next 610 2026-02-27 17:21:54.069002: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c07200 of size 2048 next 611 2026-02-27 17:21:54.069014: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c07a00 of size 2048 next 612 2026-02-27 17:21:54.069026: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c08200 of size 256 next 614 2026-02-27 17:21:54.069038: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c08300 of size 256 next 615 2026-02-27 17:21:54.069051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c08400 of size 8192 next 613 2026-02-27 17:21:54.069063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c0a400 of size 8192 next 618 2026-02-27 17:21:54.069075: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c0c400 of size 8192 next 619 2026-02-27 17:21:54.069087: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c0e400 of size 8192 next 620 2026-02-27 17:21:54.069099: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c10400 of size 8192 next 621 2026-02-27 17:21:54.069111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c12400 of size 256 next 622 2026-02-27 17:21:54.069124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c12500 of size 256 next 623 2026-02-27 17:21:54.069136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c12600 of size 8192 next 626 2026-02-27 17:21:54.069148: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c14600 of size 8192 next 627 2026-02-27 17:21:54.069160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c16600 of size 8192 next 628 2026-02-27 17:21:54.069172: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c18600 of size 8192 next 629 2026-02-27 17:21:54.069185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1a600 of size 8192 next 630 2026-02-27 17:21:54.069197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1c600 of size 2048 next 631 2026-02-27 17:21:54.069209: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1ce00 of size 2048 next 633 2026-02-27 17:21:54.069221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1d600 of size 2048 next 634 2026-02-27 17:21:54.069233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1de00 of size 2048 next 635 2026-02-27 17:21:54.069245: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1e600 of size 2048 next 636 2026-02-27 17:21:54.069258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1ee00 of size 2048 next 637 2026-02-27 17:21:54.069270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1f600 of size 2048 next 640 2026-02-27 17:21:54.069282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c1fe00 of size 2048 next 641 2026-02-27 17:21:54.069302: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c20600 of size 2048 next 642 2026-02-27 17:21:54.069315: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c20e00 of size 2048 next 643 2026-02-27 17:21:54.069327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c21600 of size 8192 next 645 2026-02-27 17:21:54.069339: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c23600 of size 8192 next 646 2026-02-27 17:21:54.069351: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c25600 of size 8192 next 647 2026-02-27 17:21:54.069364: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c27600 of size 8192 next 648 2026-02-27 17:21:54.069376: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c29600 of size 8192 next 649 2026-02-27 17:21:54.069388: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2b600 of size 2048 next 650 2026-02-27 17:21:54.069400: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2be00 of size 2048 next 651 2026-02-27 17:21:54.069413: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2c600 of size 2048 next 652 2026-02-27 17:21:54.069425: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2ce00 of size 2048 next 653 2026-02-27 17:21:54.069437: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2d600 of size 2048 next 654 2026-02-27 17:21:54.069449: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2de00 of size 2048 next 655 2026-02-27 17:21:54.069461: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2e600 of size 2048 next 657 2026-02-27 17:21:54.069474: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2ee00 of size 2048 next 658 2026-02-27 17:21:54.069486: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2f600 of size 2048 next 659 2026-02-27 17:21:54.069498: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c2fe00 of size 2048 next 660 2026-02-27 17:21:54.069510: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c30600 of size 8192 next 662 2026-02-27 17:21:54.069522: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c32600 of size 8192 next 663 2026-02-27 17:21:54.069535: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c34600 of size 8192 next 664 2026-02-27 17:21:54.069547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c36600 of size 8192 next 665 2026-02-27 17:21:54.069559: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c38600 of size 8192 next 666 2026-02-27 17:21:54.069571: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3a600 of size 1024 next 667 2026-02-27 17:21:54.069583: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3aa00 of size 1024 next 668 2026-02-27 17:21:54.069596: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3ae00 of size 1024 next 669 2026-02-27 17:21:54.069608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3b200 of size 1024 next 673 2026-02-27 17:21:54.069620: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3b600 of size 1024 next 675 2026-02-27 17:21:54.069632: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3ba00 of size 1024 next 677 2026-02-27 17:21:54.069644: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3be00 of size 1024 next 679 2026-02-27 17:21:54.069657: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3c200 of size 1024 next 681 2026-02-27 17:21:54.069669: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3c600 of size 256 next 682 2026-02-27 17:21:54.069681: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3c700 of size 256 next 683 2026-02-27 17:21:54.069693: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3c800 of size 2048 next 686 2026-02-27 17:21:54.069714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3d000 of size 256 next 690 2026-02-27 17:21:54.069726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3d100 of size 256 next 694 2026-02-27 17:21:54.069739: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3d200 of size 256 next 695 2026-02-27 17:21:54.069751: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3d300 of size 256 next 693 2026-02-27 17:21:54.069763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3d400 of size 256 next 700 2026-02-27 17:21:54.069775: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c3d500 of size 256 next 701 2026-02-27 17:21:54.069787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6b0c3d600 of size 10752 next 687 2026-02-27 17:21:54.069800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c40000 of size 256 next 688 2026-02-27 17:21:54.069812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c40100 of size 256 next 689 2026-02-27 17:21:54.069824: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6b0c40200 of size 12544 next 691 2026-02-27 17:21:54.069836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c43300 of size 12288 next 692 2026-02-27 17:21:54.069849: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6b0c46300 of size 49152 next 697 2026-02-27 17:21:54.069861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c52300 of size 24576 next 696 2026-02-27 17:21:54.069873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6b0c58300 of size 228608 next 160 2026-02-27 17:21:54.069885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0c90000 of size 589824 next 158 2026-02-27 17:21:54.069897: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0d20000 of size 524288 next 171 2026-02-27 17:21:54.069910: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0da0000 of size 256 next 671 2026-02-27 17:21:54.069922: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0da0100 of size 256 next 672 2026-02-27 17:21:54.069934: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fe6b0da0200 of size 523776 next 670 2026-02-27 17:21:54.069946: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0e20000 of size 524288 next 222 2026-02-27 17:21:54.069958: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0ea0000 of size 1048576 next 190 2026-02-27 17:21:54.069970: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b0fa0000 of size 1048576 next 189 2026-02-27 17:21:54.069983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b10a0000 of size 1048576 next 203 2026-02-27 17:21:54.069995: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b11a0000 of size 1048576 next 251 2026-02-27 17:21:54.070007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe6b12a0000 of size 1441792 next 18446744073709551615 2026-02-27 17:21:54.070019: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2026-02-27 17:21:54.070032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00000 of size 1280 next 1 2026-02-27 17:21:54.070044: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00500 of size 256 next 5 2026-02-27 17:21:54.070056: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00600 of size 256 next 8 2026-02-27 17:21:54.070069: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00700 of size 256 next 9 2026-02-27 17:21:54.070081: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00800 of size 256 next 10 2026-02-27 17:21:54.070093: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00900 of size 256 next 11 2026-02-27 17:21:54.070113: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00a00 of size 256 next 12 2026-02-27 17:21:54.070126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00b00 of size 256 next 13 2026-02-27 17:21:54.070138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00c00 of size 256 next 17 2026-02-27 17:21:54.070150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00d00 of size 256 next 19 2026-02-27 17:21:54.070163: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00e00 of size 256 next 20 2026-02-27 17:21:54.070175: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e00f00 of size 256 next 21 2026-02-27 17:21:54.070187: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01000 of size 256 next 22 2026-02-27 17:21:54.070199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01100 of size 256 next 24 2026-02-27 17:21:54.070211: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01200 of size 256 next 25 2026-02-27 17:21:54.070223: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01300 of size 256 next 23 2026-02-27 17:21:54.070236: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01400 of size 256 next 28 2026-02-27 17:21:54.070248: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01500 of size 256 next 29 2026-02-27 17:21:54.070260: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01600 of size 256 next 30 2026-02-27 17:21:54.070280: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01700 of size 256 next 31 2026-02-27 17:21:54.070296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01800 of size 256 next 33 2026-02-27 17:21:54.070308: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01900 of size 256 next 34 2026-02-27 17:21:54.070320: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01a00 of size 1024 next 32 2026-02-27 17:21:54.070333: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e01e00 of size 1024 next 37 2026-02-27 17:21:54.070345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e02200 of size 1024 next 38 2026-02-27 17:21:54.070357: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e02600 of size 1024 next 39 2026-02-27 17:21:54.070369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e02a00 of size 1024 next 40 2026-02-27 17:21:54.070381: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e02e00 of size 1024 next 41 2026-02-27 17:21:54.070393: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e03200 of size 1024 next 43 2026-02-27 17:21:54.070406: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e03600 of size 1024 next 44 2026-02-27 17:21:54.070418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e03a00 of size 1024 next 45 2026-02-27 17:21:54.070430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e03e00 of size 1024 next 46 2026-02-27 17:21:54.070442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04200 of size 256 next 48 2026-02-27 17:21:54.070454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04300 of size 256 next 49 2026-02-27 17:21:54.070466: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04400 of size 256 next 50 2026-02-27 17:21:54.070478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04500 of size 256 next 51 2026-02-27 17:21:54.070491: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04600 of size 256 next 52 2026-02-27 17:21:54.070503: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04700 of size 256 next 53 2026-02-27 17:21:54.070515: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04800 of size 256 next 56 2026-02-27 17:21:54.070537: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04900 of size 256 next 57 2026-02-27 17:21:54.070550: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04a00 of size 256 next 58 2026-02-27 17:21:54.070562: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04b00 of size 256 next 14 2026-02-27 17:21:54.070574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04c00 of size 256 next 15 2026-02-27 17:21:54.070587: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04d00 of size 256 next 16 2026-02-27 17:21:54.070599: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e04e00 of size 1024 next 60 2026-02-27 17:21:54.070611: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e05200 of size 1024 next 61 2026-02-27 17:21:54.070623: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e05600 of size 1024 next 62 2026-02-27 17:21:54.070635: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e05a00 of size 1024 next 63 2026-02-27 17:21:54.070648: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e05e00 of size 1024 next 64 2026-02-27 17:21:54.070660: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06200 of size 256 next 66 2026-02-27 17:21:54.070672: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06300 of size 256 next 67 2026-02-27 17:21:54.070684: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06400 of size 256 next 68 2026-02-27 17:21:54.070696: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06500 of size 256 next 69 2026-02-27 17:21:54.070708: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06600 of size 256 next 70 2026-02-27 17:21:54.070721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06700 of size 256 next 73 2026-02-27 17:21:54.070733: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06800 of size 256 next 74 2026-02-27 17:21:54.070745: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06900 of size 256 next 75 2026-02-27 17:21:54.070757: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06a00 of size 256 next 76 2026-02-27 17:21:54.070769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06b00 of size 256 next 77 2026-02-27 17:21:54.070781: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e06c00 of size 1024 next 79 2026-02-27 17:21:54.070794: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e07000 of size 1024 next 80 2026-02-27 17:21:54.070806: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e07400 of size 1024 next 81 2026-02-27 17:21:54.070818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e07800 of size 1024 next 82 2026-02-27 17:21:54.070830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e07c00 of size 1024 next 83 2026-02-27 17:21:54.070842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08000 of size 256 next 85 2026-02-27 17:21:54.070854: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08100 of size 256 next 86 2026-02-27 17:21:54.070867: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08200 of size 512 next 84 2026-02-27 17:21:54.070879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08400 of size 512 next 88 2026-02-27 17:21:54.070891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08600 of size 512 next 89 2026-02-27 17:21:54.070903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08800 of size 512 next 90 2026-02-27 17:21:54.070916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08a00 of size 512 next 91 2026-02-27 17:21:54.070937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08c00 of size 256 next 93 2026-02-27 17:21:54.070949: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08d00 of size 256 next 94 2026-02-27 17:21:54.070962: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e08e00 of size 512 next 92 2026-02-27 17:21:54.070974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e09000 of size 512 next 97 2026-02-27 17:21:54.070986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e09200 of size 512 next 98 2026-02-27 17:21:54.070998: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e09400 of size 512 next 99 2026-02-27 17:21:54.071010: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e09600 of size 512 next 2 2026-02-27 17:21:54.071023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e09800 of size 256 next 3 2026-02-27 17:21:54.071035: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e09900 of size 256 next 4 2026-02-27 17:21:54.071047: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e09a00 of size 16384 next 18 2026-02-27 17:21:54.071059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e0da00 of size 256 next 100 2026-02-27 17:21:54.071072: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e0db00 of size 256 next 101 2026-02-27 17:21:54.071084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e0dc00 of size 2048 next 104 2026-02-27 17:21:54.071096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e0e400 of size 2048 next 105 2026-02-27 17:21:54.071108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e0ec00 of size 2048 next 106 2026-02-27 17:21:54.071120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e0f400 of size 2048 next 107 2026-02-27 17:21:54.071132: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e0fc00 of size 2048 next 108 2026-02-27 17:21:54.071144: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e10400 of size 256 next 109 2026-02-27 17:21:54.071157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e10500 of size 256 next 110 2026-02-27 17:21:54.071169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e10600 of size 2048 next 113 2026-02-27 17:21:54.071181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e10e00 of size 2048 next 114 2026-02-27 17:21:54.071193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e11600 of size 2048 next 115 2026-02-27 17:21:54.071205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e11e00 of size 2048 next 116 2026-02-27 17:21:54.071217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e12600 of size 2048 next 6 2026-02-27 17:21:54.071230: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e12e00 of size 37632 next 7 2026-02-27 17:21:54.071242: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1c100 of size 512 next 118 2026-02-27 17:21:54.071254: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1c300 of size 512 next 119 2026-02-27 17:21:54.071267: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1c500 of size 512 next 120 2026-02-27 17:21:54.071279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1c700 of size 512 next 121 2026-02-27 17:21:54.071291: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1c900 of size 512 next 122 2026-02-27 17:21:54.071303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1cb00 of size 512 next 125 2026-02-27 17:21:54.071315: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1cd00 of size 512 next 126 2026-02-27 17:21:54.071327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1cf00 of size 512 next 127 2026-02-27 17:21:54.071348: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1d100 of size 512 next 128 2026-02-27 17:21:54.071360: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1d300 of size 512 next 129 2026-02-27 17:21:54.071373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1d500 of size 2048 next 130 2026-02-27 17:21:54.071385: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1dd00 of size 2048 next 131 2026-02-27 17:21:54.071397: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1e500 of size 2048 next 132 2026-02-27 17:21:54.071409: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1ed00 of size 2048 next 133 2026-02-27 17:21:54.071421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1f500 of size 2048 next 134 2026-02-27 17:21:54.071433: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1fd00 of size 512 next 136 2026-02-27 17:21:54.071446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e1ff00 of size 512 next 137 2026-02-27 17:21:54.071458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e20100 of size 512 next 138 2026-02-27 17:21:54.071470: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e20300 of size 512 next 139 2026-02-27 17:21:54.071482: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e20500 of size 512 next 140 2026-02-27 17:21:54.071494: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e20700 of size 512 next 141 2026-02-27 17:21:54.071507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e20900 of size 512 next 143 2026-02-27 17:21:54.071519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e20b00 of size 512 next 144 2026-02-27 17:21:54.071531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e20d00 of size 512 next 145 2026-02-27 17:21:54.071543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e20f00 of size 512 next 146 2026-02-27 17:21:54.071555: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e21100 of size 2048 next 147 2026-02-27 17:21:54.071568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e21900 of size 2048 next 148 2026-02-27 17:21:54.071580: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e22100 of size 2048 next 149 2026-02-27 17:21:54.071592: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e22900 of size 2048 next 150 2026-02-27 17:21:54.071604: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e23100 of size 2048 next 151 2026-02-27 17:21:54.071616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e23900 of size 512 next 153 2026-02-27 17:21:54.071628: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e23b00 of size 512 next 154 2026-02-27 17:21:54.071641: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e23d00 of size 512 next 155 2026-02-27 17:21:54.071653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e23f00 of size 512 next 156 2026-02-27 17:21:54.071665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e24100 of size 512 next 157 2026-02-27 17:21:54.071677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e24300 of size 512 next 161 2026-02-27 17:21:54.071689: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e24500 of size 512 next 162 2026-02-27 17:21:54.071701: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e24700 of size 512 next 163 2026-02-27 17:21:54.071714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e24900 of size 512 next 164 2026-02-27 17:21:54.071726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e24b00 of size 512 next 165 2026-02-27 17:21:54.071746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e24d00 of size 2048 next 166 2026-02-27 17:21:54.071759: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e25500 of size 2048 next 167 2026-02-27 17:21:54.071771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e25d00 of size 2048 next 168 2026-02-27 17:21:54.071783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e26500 of size 2048 next 169 2026-02-27 17:21:54.071795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e26d00 of size 2048 next 170 2026-02-27 17:21:54.071807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e27500 of size 1024 next 172 2026-02-27 17:21:54.071820: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e27900 of size 1024 next 173 2026-02-27 17:21:54.071832: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e27d00 of size 1024 next 174 2026-02-27 17:21:54.071844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e28100 of size 1024 next 175 2026-02-27 17:21:54.071856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e28500 of size 1024 next 176 2026-02-27 17:21:54.071868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e28900 of size 256 next 178 2026-02-27 17:21:54.071881: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e28a00 of size 256 next 179 2026-02-27 17:21:54.071893: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e28b00 of size 1024 next 177 2026-02-27 17:21:54.071905: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e28f00 of size 1024 next 182 2026-02-27 17:21:54.071917: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e29300 of size 1024 next 183 2026-02-27 17:21:54.071929: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e29700 of size 1024 next 184 2026-02-27 17:21:54.071941: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e29b00 of size 1024 next 185 2026-02-27 17:21:54.071954: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e29f00 of size 256 next 187 2026-02-27 17:21:54.071966: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e2a000 of size 256 next 188 2026-02-27 17:21:54.071978: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e2a100 of size 4096 next 186 2026-02-27 17:21:54.071990: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e2b100 of size 4096 next 42 2026-02-27 17:21:54.072002: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e2c100 of size 65536 next 36 2026-02-27 17:21:54.072014: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e3c100 of size 65536 next 35 2026-02-27 17:21:54.072027: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e4c100 of size 4096 next 235 2026-02-27 17:21:54.072039: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e4d100 of size 4096 next 236 2026-02-27 17:21:54.072051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e4e100 of size 4096 next 237 2026-02-27 17:21:54.072063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e4f100 of size 4096 next 238 2026-02-27 17:21:54.072075: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e50100 of size 1024 next 239 2026-02-27 17:21:54.072087: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e50500 of size 1024 next 240 2026-02-27 17:21:54.072099: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e50900 of size 1024 next 241 2026-02-27 17:21:54.072112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e50d00 of size 1024 next 242 2026-02-27 17:21:54.072124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e51100 of size 1024 next 243 2026-02-27 17:21:54.072136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e51500 of size 1024 next 246 2026-02-27 17:21:54.072156: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e51900 of size 1024 next 247 2026-02-27 17:21:54.072169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e51d00 of size 1024 next 248 2026-02-27 17:21:54.072181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e52100 of size 1024 next 249 2026-02-27 17:21:54.072193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e52500 of size 1024 next 250 2026-02-27 17:21:54.072205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e52900 of size 4096 next 252 2026-02-27 17:21:54.072217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e53900 of size 4096 next 253 2026-02-27 17:21:54.072230: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e54900 of size 4096 next 254 2026-02-27 17:21:54.072242: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e55900 of size 4096 next 255 2026-02-27 17:21:54.072254: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e56900 of size 4096 next 256 2026-02-27 17:21:54.072266: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e57900 of size 1024 next 257 2026-02-27 17:21:54.072278: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e57d00 of size 1024 next 258 2026-02-27 17:21:54.072290: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e58100 of size 1024 next 259 2026-02-27 17:21:54.072303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e58500 of size 1024 next 260 2026-02-27 17:21:54.072315: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e58900 of size 1024 next 261 2026-02-27 17:21:54.072327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e58d00 of size 1024 next 264 2026-02-27 17:21:54.072339: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e59100 of size 1024 next 265 2026-02-27 17:21:54.072351: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e59500 of size 1024 next 266 2026-02-27 17:21:54.072363: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e59900 of size 1024 next 267 2026-02-27 17:21:54.072376: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e59d00 of size 1024 next 268 2026-02-27 17:21:54.072388: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5a100 of size 4096 next 270 2026-02-27 17:21:54.072400: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5b100 of size 4096 next 271 2026-02-27 17:21:54.072412: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5c100 of size 4096 next 272 2026-02-27 17:21:54.072424: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5d100 of size 4096 next 273 2026-02-27 17:21:54.072436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5e100 of size 4096 next 274 2026-02-27 17:21:54.072449: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5f100 of size 1024 next 275 2026-02-27 17:21:54.072461: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5f500 of size 1024 next 276 2026-02-27 17:21:54.072473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5f900 of size 1024 next 277 2026-02-27 17:21:54.072485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e5fd00 of size 1024 next 278 2026-02-27 17:21:54.072497: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e60100 of size 1024 next 279 2026-02-27 17:21:54.072509: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e60500 of size 1024 next 282 2026-02-27 17:21:54.072522: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e60900 of size 1024 next 283 2026-02-27 17:21:54.072534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e60d00 of size 1024 next 284 2026-02-27 17:21:54.072554: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e61100 of size 1024 next 285 2026-02-27 17:21:54.072567: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e61500 of size 1024 next 286 2026-02-27 17:21:54.072579: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e61900 of size 4096 next 288 2026-02-27 17:21:54.072591: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e62900 of size 6144 next 27 2026-02-27 17:21:54.072603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e64100 of size 147456 next 26 2026-02-27 17:21:54.072616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e88100 of size 65536 next 47 2026-02-27 17:21:54.072628: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6e98100 of size 65536 next 59 2026-02-27 17:21:54.072640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6ea8100 of size 65536 next 65 2026-02-27 17:21:54.072653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fe7e6eb8100 of size 294656 next 18446744073709551615 2026-02-27 17:21:54.072664: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2026-02-27 17:21:54.072680: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 77 Chunks of size 256 totalling 19.2KiB 2026-02-27 17:21:54.072694: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2026-02-27 17:21:54.072708: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 258 Chunks of size 1024 totalling 258.0KiB 2026-02-27 17:21:54.072721: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2026-02-27 17:21:54.072735: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 56 Chunks of size 2048 totalling 112.0KiB 2026-02-27 17:21:54.072748: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 117 Chunks of size 4096 totalling 468.0KiB 2026-02-27 17:21:54.072762: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 6144 totalling 12.0KiB 2026-02-27 17:21:54.072775: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2026-02-27 17:21:54.072788: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 20 Chunks of size 8192 totalling 160.0KiB 2026-02-27 17:21:54.072802: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 12288 totalling 12.0KiB 2026-02-27 17:21:54.072815: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2026-02-27 17:21:54.072829: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 24576 totalling 24.0KiB 2026-02-27 17:21:54.072842: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2026-02-27 17:21:54.072856: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2026-02-27 17:21:54.072869: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2026-02-27 17:21:54.072882: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2026-02-27 17:21:54.072896: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 5 Chunks of size 262144 totalling 1.25MiB 2026-02-27 17:21:54.072909: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2026-02-27 17:21:54.072923: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2026-02-27 17:21:54.072936: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 524288 totalling 1.50MiB 2026-02-27 17:21:54.072949: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2026-02-27 17:21:54.072962: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2026-02-27 17:21:54.072985: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1048576 totalling 25.00MiB 2026-02-27 17:21:54.072999: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 19 Chunks of size 1310720 totalling 23.75MiB 2026-02-27 17:21:54.073012: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB 2026-02-27 17:21:54.073025: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB 2026-02-27 17:21:54.073038: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 2097152 totalling 4.00MiB 2026-02-27 17:21:54.073052: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 26 Chunks of size 2359296 totalling 58.50MiB 2026-02-27 17:21:54.073065: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 3932160 totalling 3.75MiB 2026-02-27 17:21:54.073078: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 5 Chunks of size 4194304 totalling 20.00MiB 2026-02-27 17:21:54.073091: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 4718592 totalling 4.50MiB 2026-02-27 17:21:54.073104: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 6291456 totalling 6.00MiB 2026-02-27 17:21:54.073118: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 8388608 totalling 8.00MiB 2026-02-27 17:21:54.073131: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 9437184 totalling 27.00MiB 2026-02-27 17:21:54.073144: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 51380224 totalling 49.00MiB 2026-02-27 17:21:54.073158: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 100335616 totalling 95.69MiB 2026-02-27 17:21:54.073171: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 336.45MiB 2026-02-27 17:21:54.073184: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 419102720 memory_limit_: 419102720 available bytes: 0 curr_region_allocation_bytes_: 536870912 2026-02-27 17:21:54.073201: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 419102720 InUse: 352791040 MaxInUse: 352791296 NumAllocs: 2327 MaxAllocSize: 100335616 2026-02-27 17:21:54.073233: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *************************xxxxxxxxxxxx*************_*****_**_________******************************** 2026-02-27 17:21:54.073287: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[7,7,256,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[7,7,256,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] we want to redo the detection max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 ERROR : 'int' object is not subscriptable reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: "'int' object is not subscriptable" #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : sam list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.18502235412597656 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:sam Fri Feb 27 17:21:54 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step sam ! Inside sam : nb paths : 1 ERROR in datou_step_exec, will save and exit ! CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2340, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2441, in datou_step_exec return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 367, in datou_step_sam sam.to(device=device) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to return self._apply(convert) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 602, in _apply param_applied = fn(param) File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) [1189321094] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.017512083053588867 save_final ERROR in last step sam, CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. time spend for datou_step_exec : 9.522823095321655 time spend to save output : 0.019572973251342773 total time spend for step 0 : 9.542396068572998 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : None ERROR nb objects espect : 98 nb_objects detect : 0 ERROR sam FAILED ############################### TEST frcnn ################################ Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : frcnn list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.11775851249694824 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:frcnn Fri Feb 27 17:22:04 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step Faster rcnn ! To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : [] file manque in s3 : ['caffemodel', 'test.prototxt'] WARNING: Logging before InitGoogleLogging() is written to STDERR E0227 17:22:06.531243 1427270 common.cpp:114] Cannot create Cublas handle. Cublas won't be available. E0227 17:22:06.537468 1427270 common.cpp:121] Cannot create Curand generator. Curand won't be available. [libprotobuf ERROR google/protobuf/text_format.cc:307] Error parsing text-format caffe.NetParameter: 325:21: Message type "caffe.LayerParameter" has no field named "roi_pooling_param". F0227 17:22:06.543825 1427270 upgrade_proto.cpp:90] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: /data/models_weight/detection_plaque_valcor_010622/test.prototxt *** Check failure stack trace: *** Aborted (core dumped) No data to report. No data to report. ret : 34304 command : coverage3 html -i --omit=/usr/local/lib/python3.8/dist-packages/*,/home/admin/.local/lib/python3.8/site-packages/*,/usr/lib/python3/dist-packages/* -d htmlcov ret : 256 command : coverage3 report -i -m ret : 256 15.17user 9.53system 1:41.22elapsed 24%CPU (0avgtext+0avgdata 2414140maxresident)k 2449224inputs+3352outputs (6138major+1150717minor)pagefaults 0swaps