python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120
 import MySQLdb succeeded 
Import error (python version)
python version = 3
warning , we can't find thcl infos in json_data
warning , we can't find pdt infos in json_data
 list_job_run_as_list :  ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel']
python version used : 3
 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')]


#&_# BEGIN OF TEST : tests/mask_test #&_# 


/home/admin/workarea/git/Velours/python/tests/mask_test.py
<module 'tests.mask_test' from '/home/admin/workarea/git/Velours/python/tests/mask_test.py'>
Test mask-detection 
python version used : 3

############################### TEST memory used  ################################
free memory at begining : 
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 1746

run mask_detect

Inside batchDatouExec : verbose : False
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
List Step Type Loaded in datou : mask_detect
list_input_json : []
origin
BFwe have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.11851859092712402
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Thu Apr 24 15:35:28 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3000
l 3632 free memory gpu now : 1746
wait 20 seconds
l 3637 free memory gpu now : 1746  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
/home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
2025-04-24 15:35:51.904355: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-04-24 15:35:51.912819: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz
2025-04-24 15:35:51.914533: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc48c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-04-24 15:35:51.914589: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-04-24 15:35:51.918135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-04-24 15:35:52.138211: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xa8f4b20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-04-24 15:35:52.138258: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-04-24 15:35:52.139381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-04-24 15:35:52.140136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-04-24 15:35:52.143881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-04-24 15:35:52.146362: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-04-24 15:35:52.147868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-04-24 15:35:52.151657: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-04-24 15:35:52.153333: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-04-24 15:35:52.159451: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-04-24 15:35:52.160707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-04-24 15:35:52.160799: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-04-24 15:35:52.161386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-04-24 15:35:52.161404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-04-24 15:35:52.161414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-04-24 15:35:52.162394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1369 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
To do loadFromThcl(), then load ParamDescType : thcl454
 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3473
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
2025-04-24 15:35:52.815654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-04-24 15:35:52.815764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-04-24 15:35:52.815797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-04-24 15:35:52.815828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-04-24 15:35:52.815862: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-04-24 15:35:52.815902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-04-24 15:35:52.815953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-04-24 15:35:52.815985: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-04-24 15:35:52.816955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-04-24 15:35:52.817897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-04-24 15:35:52.817926: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-04-24 15:35:52.817941: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-04-24 15:35:52.817953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-04-24 15:35:52.817966: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-04-24 15:35:52.817979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-04-24 15:35:52.817993: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-04-24 15:35:52.818007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-04-24 15:35:52.818699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-04-24 15:35:52.818729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-04-24 15:35:52.818737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-04-24 15:35:52.818744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-04-24 15:35:52.819551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1369 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
Using TensorFlow backend.
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


model_param file didn't exist 
model_name : mask_coco_origin
model_type : mask_rcnn
list file need :
['mask_model.h5']
file exist in s3 :
['mask_model.h5']
file manque in s3 : 
[]
2025-04-24 15:36:00.009081: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-04-24 15:36:00.235851: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-04-24 15:36:01.757137: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.757216: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.763912: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.763952: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.812457: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.812503: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.854921: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.854987: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.907442: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.907482: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-04-24 15:36:01.939979: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2025-04-24 15:36:01.956717: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.957814: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 772.99M (810536192 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.958793: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 695.69M (729482752 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.959848: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 626.12M (656534528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.960901: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.963727: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.964466: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.972030: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.972674: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.978707: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.979317: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.980944: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:01.981565: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.015620: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.016391: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.017104: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.017816: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.021434: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.021947: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.037642: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.038144: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.038609: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.039101: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.051485: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.052002: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.052488: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.052981: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.061112: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.061604: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.073731: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.074224: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.078265: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.078734: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.099610: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.100123: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.100621: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.101089: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.101556: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.102022: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.143217: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.143284: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2025-04-24 15:36:02.144292: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.145171: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.152659: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.153581: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.161693: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.162279: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.179274: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.179966: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.180607: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.181228: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.185386: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.185872: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.186367: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.186868: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.187913: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.198058: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.198529: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.208841: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.209336: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.209813: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.210273: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.210744: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-04-24 15:36:02.211247: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 858.88M (900595712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
local folder  : /data/models_weight/mask_coco_origin


/data/models_weight/mask_coco_origin/mask_model.h5
size_local : 257557808 size in s3 : 257557808
create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17
mask_model.h5 already exist and didn't need to update
 list_images length : 1
NEW PHOTO
Processing 1 images
image                    shape: (480, 640, 3)         min:    0.00000  max:  255.00000
molded_images            shape: (1, 640, 640, 3)      min: -123.70000  max:  151.10000
image_metas              shape: (1, 89)               min:    0.00000  max:  640.00000
nb d'objets trouves : 5
Detection mask done !
Trying to reset tf kernel
1173737
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 553
tf kernel not reseted
sub process len(results) : 1 len(list_Values) 0
None
max_time_sub_proc : 3600
parent process len(results) : 1 len(list_Values) 0
process is alive
finish correctly or not : True

 after detect 

begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 1746
list_Values should be empty []
To do loadFromThcl(), then load ParamDescType : thcl454
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.16195130348205566
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Thu Apr 24 15:36:08 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3000
l 3632 free memory gpu now : 1491
wait 20 seconds
l 3637 free memory gpu now : 1491  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
2025-04-24 15:36:31.772149: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-04-24 15:36:31.799173: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz
2025-04-24 15:36:31.801505: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc48c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-04-24 15:36:31.801582: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-04-24 15:36:31.806235: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-04-24 15:36:31.943200: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xb4e5d70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-04-24 15:36:31.943247: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-04-24 15:36:31.944077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-04-24 15:36:31.944509: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-04-24 15:36:31.947062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-04-24 15:36:31.949861: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-04-24 15:36:31.950327: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-04-24 15:36:31.953010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-04-24 15:36:31.954373: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-04-24 15:36:31.959844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-04-24 15:36:31.960994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-04-24 15:36:31.961146: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-04-24 15:36:31.961902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-04-24 15:36:31.961932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-04-24 15:36:31.961945: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-04-24 15:36:31.963067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 176 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

2025-04-24 15:36:32.064960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-04-24 15:36:32.065144: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-04-24 15:36:32.065177: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-04-24 15:36:32.065206: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-04-24 15:36:32.065234: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-04-24 15:36:32.065260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-04-24 15:36:32.065288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-04-24 15:36:32.065315: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-04-24 15:36:32.066283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-04-24 15:36:32.067583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-04-24 15:36:32.067702: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-04-24 15:36:32.067729: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-04-24 15:36:32.067753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-04-24 15:36:32.067776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-04-24 15:36:32.067813: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-04-24 15:36:32.067837: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-04-24 15:36:32.067861: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-04-24 15:36:32.068810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-04-24 15:36:32.068856: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-04-24 15:36:32.068870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-04-24 15:36:32.068882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-04-24 15:36:32.069884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 176 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
2025-04-24 15:36:44.427297: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 9.00MiB (rounded to 9437184)
Current allocation summary follows.
2025-04-24 15:36:44.427369: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc
2025-04-24 15:36:44.427385: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): 	Total Chunks: 65, Chunks in use: 65. 16.2KiB allocated for chunks. 16.2KiB in use in bin. 8.9KiB client-requested in use in bin.
2025-04-24 15:36:44.427398: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): 	Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin.
2025-04-24 15:36:44.427409: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): 	Total Chunks: 251, Chunks in use: 251. 251.2KiB allocated for chunks. 251.2KiB in use in bin. 251.0KiB client-requested in use in bin.
2025-04-24 15:36:44.427448: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): 	Total Chunks: 40, Chunks in use: 40. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin.
2025-04-24 15:36:44.427461: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): 	Total Chunks: 120, Chunks in use: 120. 487.5KiB allocated for chunks. 487.5KiB in use in bin. 480.0KiB client-requested in use in bin.
2025-04-24 15:36:44.427472: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): 	Total Chunks: 10, Chunks in use: 10. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin.
2025-04-24 15:36:44.427483: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): 	Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2025-04-24 15:36:44.427494: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): 	Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin.
2025-04-24 15:36:44.427505: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): 	Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin.
2025-04-24 15:36:44.427516: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): 	Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin.
2025-04-24 15:36:44.427527: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): 	Total Chunks: 8, Chunks in use: 7. 2.35MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin.
2025-04-24 15:36:44.427557: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): 	Total Chunks: 7, Chunks in use: 6. 4.39MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin.
2025-04-24 15:36:44.427568: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): 	Total Chunks: 48, Chunks in use: 45. 54.12MiB allocated for chunks. 50.62MiB in use in bin. 45.00MiB client-requested in use in bin.
2025-04-24 15:36:44.427579: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): 	Total Chunks: 28, Chunks in use: 25. 64.25MiB allocated for chunks. 55.75MiB in use in bin. 55.75MiB client-requested in use in bin.
2025-04-24 15:36:44.427590: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): 	Total Chunks: 4, Chunks in use: 2. 18.00MiB allocated for chunks. 8.00MiB in use in bin. 8.00MiB client-requested in use in bin.
2025-04-24 15:36:44.427601: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): 	Total Chunks: 3, Chunks in use: 2. 31.88MiB allocated for chunks. 23.88MiB in use in bin. 17.00MiB client-requested in use in bin.
2025-04-24 15:36:44.427611: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-04-24 15:36:44.427621: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-04-24 15:36:44.427630: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-04-24 15:36:44.427640: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-04-24 15:36:44.427649: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-04-24 15:36:44.427659: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 9.00MiB was 8.00MiB, Chunk State: 
2025-04-24 15:36:44.427676: I tensorflow/core/common_runtime/bfc_allocator.cc:963]   Size: 8.00MiB | Requested Size: 8.00MiB | in_use: 0 | bin_num: 15, prev:   Size: 9.00MiB | Requested Size: 9.00MiB | in_use: 1 | bin_num: -1, next:   Size: 14.88MiB | Requested Size: 8.00MiB | in_use: 1 | bin_num: -1
2025-04-24 15:36:44.427685: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 52297728
2025-04-24 15:36:44.427696: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2ac000000 of size 4194304 next 631
2025-04-24 15:36:44.427706: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2ac400000 of size 4194304 next 617
2025-04-24 15:36:44.427714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2ac800000 of size 4194304 next 616
2025-04-24 15:36:44.427724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2acc00000 of size 6291456 next 608
2025-04-24 15:36:44.427735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2ad200000 of size 9437184 next 607
2025-04-24 15:36:44.427744: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2adb00000 of size 8388608 next 625
2025-04-24 15:36:44.427753: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2ae300000 of size 15597568 next 18446744073709551615
2025-04-24 15:36:44.427761: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 67108864
2025-04-24 15:36:44.427770: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b0000000 of size 2359296 next 370
2025-04-24 15:36:44.427779: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b0240000 of size 1048576 next 394
2025-04-24 15:36:44.427787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2b0340000 of size 1310720 next 388
2025-04-24 15:36:44.427802: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b0480000 of size 2359296 next 387
2025-04-24 15:36:44.427811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b06c0000 of size 1048576 next 412
2025-04-24 15:36:44.427819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b07c0000 of size 1310720 next 406
2025-04-24 15:36:44.427828: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b0900000 of size 2359296 next 405
2025-04-24 15:36:44.427836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b0b40000 of size 1048576 next 430
2025-04-24 15:36:44.427844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b0c40000 of size 1310720 next 424
2025-04-24 15:36:44.427853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b0d80000 of size 2359296 next 423
2025-04-24 15:36:44.427861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b0fc0000 of size 1048576 next 447
2025-04-24 15:36:44.427869: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b10c0000 of size 1310720 next 441
2025-04-24 15:36:44.427878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b1200000 of size 2359296 next 440
2025-04-24 15:36:44.427886: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b1440000 of size 1048576 next 465
2025-04-24 15:36:44.427895: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b1540000 of size 1310720 next 459
2025-04-24 15:36:44.427903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b1680000 of size 2359296 next 458
2025-04-24 15:36:44.427911: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b18c0000 of size 1048576 next 483
2025-04-24 15:36:44.427919: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b19c0000 of size 1310720 next 477
2025-04-24 15:36:44.427928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b1b00000 of size 2359296 next 476
2025-04-24 15:36:44.427936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b1d40000 of size 1048576 next 501
2025-04-24 15:36:44.427944: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b1e40000 of size 1310720 next 495
2025-04-24 15:36:44.427953: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b1f80000 of size 2359296 next 494
2025-04-24 15:36:44.427961: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b21c0000 of size 1048576 next 519
2025-04-24 15:36:44.427969: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b22c0000 of size 1310720 next 513
2025-04-24 15:36:44.427978: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b2400000 of size 2359296 next 512
2025-04-24 15:36:44.427986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b2640000 of size 1048576 next 537
2025-04-24 15:36:44.427994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b2740000 of size 1310720 next 531
2025-04-24 15:36:44.428003: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b2880000 of size 2359296 next 530
2025-04-24 15:36:44.428011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b2ac0000 of size 1048576 next 555
2025-04-24 15:36:44.428019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b2bc0000 of size 1310720 next 549
2025-04-24 15:36:44.428028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b2d00000 of size 2359296 next 548
2025-04-24 15:36:44.428036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b2f40000 of size 1048576 next 573
2025-04-24 15:36:44.428044: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b3040000 of size 1310720 next 567
2025-04-24 15:36:44.428052: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b3180000 of size 2359296 next 566
2025-04-24 15:36:44.428067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b33c0000 of size 1048576 next 591
2025-04-24 15:36:44.428076: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2b34c0000 of size 1310720 next 585
2025-04-24 15:36:44.428084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b3600000 of size 2359296 next 584
2025-04-24 15:36:44.428092: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2b3840000 of size 2097152 next 597
2025-04-24 15:36:44.428101: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b3a40000 of size 2097152 next 596
2025-04-24 15:36:44.428109: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2b3c40000 of size 3932160 next 18446744073709551615
2025-04-24 15:36:44.428118: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432
2025-04-24 15:36:44.428127: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4000000 of size 2359296 next 245
2025-04-24 15:36:44.428135: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4240000 of size 1048576 next 269
2025-04-24 15:36:44.428143: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4340000 of size 1310720 next 263
2025-04-24 15:36:44.428152: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4480000 of size 2359296 next 262
2025-04-24 15:36:44.428160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b46c0000 of size 1048576 next 287
2025-04-24 15:36:44.428168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b47c0000 of size 1310720 next 281
2025-04-24 15:36:44.428177: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4900000 of size 2359296 next 280
2025-04-24 15:36:44.428185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4b40000 of size 1048576 next 304
2025-04-24 15:36:44.428193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4c40000 of size 1310720 next 298
2025-04-24 15:36:44.428202: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4d80000 of size 2359296 next 297
2025-04-24 15:36:44.428210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b4fc0000 of size 1048576 next 322
2025-04-24 15:36:44.428218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b50c0000 of size 1310720 next 316
2025-04-24 15:36:44.428226: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b5200000 of size 2359296 next 315
2025-04-24 15:36:44.428234: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b5440000 of size 1048576 next 340
2025-04-24 15:36:44.428243: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b5540000 of size 1310720 next 334
2025-04-24 15:36:44.428251: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b5680000 of size 2359296 next 333
2025-04-24 15:36:44.428259: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b58c0000 of size 1048576 next 358
2025-04-24 15:36:44.428268: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b59c0000 of size 1310720 next 352
2025-04-24 15:36:44.428276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b5b00000 of size 2359296 next 351
2025-04-24 15:36:44.428284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b5d40000 of size 1048576 next 376
2025-04-24 15:36:44.428293: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b5e40000 of size 1835008 next 18446744073709551615
2025-04-24 15:36:44.428301: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216
2025-04-24 15:36:44.428309: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b6000000 of size 2359296 next 181
2025-04-24 15:36:44.428318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b6240000 of size 2097152 next 196
2025-04-24 15:36:44.428326: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b6440000 of size 1048576 next 216
2025-04-24 15:36:44.428340: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b6540000 of size 1310720 next 209
2025-04-24 15:36:44.428349: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b6680000 of size 2359296 next 208
2025-04-24 15:36:44.428357: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b68c0000 of size 1048576 next 234
2025-04-24 15:36:44.428365: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b69c0000 of size 1310720 next 228
2025-04-24 15:36:44.428373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b6b00000 of size 2359296 next 227
2025-04-24 15:36:44.428381: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2b6d40000 of size 2883584 next 18446744073709551615
2025-04-24 15:36:44.428390: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152
2025-04-24 15:36:44.428398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8600000 of size 147456 next 55
2025-04-24 15:36:44.428406: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8624000 of size 65536 next 78
2025-04-24 15:36:44.428415: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8634000 of size 4096 next 191
2025-04-24 15:36:44.428424: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8635000 of size 4096 next 192
2025-04-24 15:36:44.428432: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8636000 of size 4096 next 193
2025-04-24 15:36:44.428440: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8637000 of size 256 next 194
2025-04-24 15:36:44.428448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8637100 of size 256 next 195
2025-04-24 15:36:44.428457: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8637200 of size 4096 next 197
2025-04-24 15:36:44.428465: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8638200 of size 4096 next 198
2025-04-24 15:36:44.428474: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8639200 of size 4096 next 199
2025-04-24 15:36:44.428482: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863a200 of size 4096 next 200
2025-04-24 15:36:44.428490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863b200 of size 4096 next 201
2025-04-24 15:36:44.428498: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863c200 of size 1024 next 202
2025-04-24 15:36:44.428518: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863c600 of size 1024 next 204
2025-04-24 15:36:44.428527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863ca00 of size 1024 next 205
2025-04-24 15:36:44.428536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863ce00 of size 1024 next 206
2025-04-24 15:36:44.428544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863d200 of size 1024 next 207
2025-04-24 15:36:44.428552: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863d600 of size 1024 next 210
2025-04-24 15:36:44.428561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863da00 of size 1024 next 211
2025-04-24 15:36:44.428569: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863de00 of size 1024 next 212
2025-04-24 15:36:44.428577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863e200 of size 1024 next 213
2025-04-24 15:36:44.428585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863e600 of size 1024 next 214
2025-04-24 15:36:44.428594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863ea00 of size 4096 next 215
2025-04-24 15:36:44.428602: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b863fa00 of size 4096 next 217
2025-04-24 15:36:44.428611: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8640a00 of size 4096 next 218
2025-04-24 15:36:44.428627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8641a00 of size 4096 next 219
2025-04-24 15:36:44.428636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8642a00 of size 4096 next 220
2025-04-24 15:36:44.428644: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8643a00 of size 1024 next 221
2025-04-24 15:36:44.428653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8643e00 of size 1024 next 223
2025-04-24 15:36:44.428661: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8644200 of size 1024 next 224
2025-04-24 15:36:44.428669: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8644600 of size 1024 next 225
2025-04-24 15:36:44.428677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8644a00 of size 1024 next 226
2025-04-24 15:36:44.428686: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8644e00 of size 1024 next 229
2025-04-24 15:36:44.428694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8645200 of size 1024 next 230
2025-04-24 15:36:44.428702: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8645600 of size 1024 next 231
2025-04-24 15:36:44.428710: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8645a00 of size 1024 next 232
2025-04-24 15:36:44.428718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8645e00 of size 1024 next 233
2025-04-24 15:36:44.428727: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8646200 of size 7680 next 72
2025-04-24 15:36:44.428736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8648000 of size 147456 next 71
2025-04-24 15:36:44.428744: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b866c000 of size 131072 next 87
2025-04-24 15:36:44.428752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b868c000 of size 4096 next 289
2025-04-24 15:36:44.428760: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b868d000 of size 4096 next 290
2025-04-24 15:36:44.428769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b868e000 of size 4096 next 291
2025-04-24 15:36:44.428777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b868f000 of size 1024 next 292
2025-04-24 15:36:44.428786: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b868f400 of size 1024 next 293
2025-04-24 15:36:44.428795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b868f800 of size 1024 next 294
2025-04-24 15:36:44.428803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b868fc00 of size 1024 next 295
2025-04-24 15:36:44.428811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8690000 of size 1024 next 296
2025-04-24 15:36:44.428819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8690400 of size 1024 next 299
2025-04-24 15:36:44.428828: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8690800 of size 1024 next 300
2025-04-24 15:36:44.428836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8690c00 of size 1024 next 301
2025-04-24 15:36:44.428844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8691000 of size 1024 next 302
2025-04-24 15:36:44.428852: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8691400 of size 1024 next 303
2025-04-24 15:36:44.428861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8691800 of size 4096 next 305
2025-04-24 15:36:44.428869: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8692800 of size 4096 next 306
2025-04-24 15:36:44.428877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8693800 of size 4096 next 307
2025-04-24 15:36:44.428885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8694800 of size 4096 next 308
2025-04-24 15:36:44.428900: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8695800 of size 4096 next 309
2025-04-24 15:36:44.428908: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8696800 of size 1024 next 310
2025-04-24 15:36:44.428917: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8696c00 of size 1024 next 311
2025-04-24 15:36:44.428925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8697000 of size 1024 next 312
2025-04-24 15:36:44.428933: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8697400 of size 1024 next 313
2025-04-24 15:36:44.428942: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8697800 of size 1024 next 314
2025-04-24 15:36:44.428951: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8697c00 of size 1024 next 317
2025-04-24 15:36:44.428959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8698000 of size 1024 next 318
2025-04-24 15:36:44.428989: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8698400 of size 1024 next 319
2025-04-24 15:36:44.428998: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8698800 of size 1024 next 320
2025-04-24 15:36:44.429006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8698c00 of size 1024 next 321
2025-04-24 15:36:44.429014: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8699000 of size 4096 next 323
2025-04-24 15:36:44.429023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869a000 of size 4096 next 324
2025-04-24 15:36:44.429032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869b000 of size 4096 next 325
2025-04-24 15:36:44.429042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869c000 of size 4096 next 326
2025-04-24 15:36:44.429054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869d000 of size 4096 next 327
2025-04-24 15:36:44.429066: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869e000 of size 1024 next 328
2025-04-24 15:36:44.429079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869e400 of size 1024 next 329
2025-04-24 15:36:44.429091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869e800 of size 1024 next 330
2025-04-24 15:36:44.429104: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869ec00 of size 1024 next 331
2025-04-24 15:36:44.429116: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869f000 of size 1024 next 332
2025-04-24 15:36:44.429129: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869f400 of size 1024 next 335
2025-04-24 15:36:44.429141: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869f800 of size 1024 next 336
2025-04-24 15:36:44.429154: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b869fc00 of size 1024 next 337
2025-04-24 15:36:44.429166: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a0000 of size 1024 next 338
2025-04-24 15:36:44.429177: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a0400 of size 1024 next 339
2025-04-24 15:36:44.429187: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a0800 of size 4096 next 341
2025-04-24 15:36:44.429196: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a1800 of size 4096 next 342
2025-04-24 15:36:44.429205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a2800 of size 4096 next 343
2025-04-24 15:36:44.429213: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a3800 of size 4096 next 344
2025-04-24 15:36:44.429221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a4800 of size 4096 next 345
2025-04-24 15:36:44.429229: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a5800 of size 1024 next 346
2025-04-24 15:36:44.429246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a5c00 of size 1024 next 347
2025-04-24 15:36:44.429255: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a6000 of size 1024 next 348
2025-04-24 15:36:44.429263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a6400 of size 1024 next 349
2025-04-24 15:36:44.429271: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a6800 of size 1024 next 350
2025-04-24 15:36:44.429279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a6c00 of size 1024 next 353
2025-04-24 15:36:44.429287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a7000 of size 1024 next 354
2025-04-24 15:36:44.429298: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a7400 of size 1024 next 355
2025-04-24 15:36:44.429308: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a7800 of size 1024 next 356
2025-04-24 15:36:44.429316: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a7c00 of size 1024 next 357
2025-04-24 15:36:44.429325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a8000 of size 4096 next 359
2025-04-24 15:36:44.429334: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86a9000 of size 4096 next 360
2025-04-24 15:36:44.429343: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86aa000 of size 4096 next 361
2025-04-24 15:36:44.429352: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ab000 of size 4096 next 362
2025-04-24 15:36:44.429360: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ac000 of size 4096 next 363
2025-04-24 15:36:44.429369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ad000 of size 1024 next 364
2025-04-24 15:36:44.429378: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ad400 of size 1024 next 365
2025-04-24 15:36:44.429387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ad800 of size 1024 next 366
2025-04-24 15:36:44.429395: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86adc00 of size 1024 next 367
2025-04-24 15:36:44.429404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ae000 of size 1024 next 368
2025-04-24 15:36:44.429412: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ae400 of size 1024 next 371
2025-04-24 15:36:44.429421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ae800 of size 1024 next 372
2025-04-24 15:36:44.429430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86aec00 of size 1024 next 373
2025-04-24 15:36:44.429438: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86af000 of size 1024 next 374
2025-04-24 15:36:44.429447: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86af400 of size 1024 next 375
2025-04-24 15:36:44.429456: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86af800 of size 4096 next 377
2025-04-24 15:36:44.429464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b0800 of size 4096 next 378
2025-04-24 15:36:44.429473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b1800 of size 4096 next 379
2025-04-24 15:36:44.429507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b2800 of size 4096 next 380
2025-04-24 15:36:44.429517: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b3800 of size 4096 next 381
2025-04-24 15:36:44.429526: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b4800 of size 1024 next 382
2025-04-24 15:36:44.429535: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b4c00 of size 1024 next 383
2025-04-24 15:36:44.429544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b5000 of size 1024 next 384
2025-04-24 15:36:44.429553: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b5400 of size 1024 next 385
2025-04-24 15:36:44.429568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b5800 of size 1024 next 386
2025-04-24 15:36:44.429577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b5c00 of size 1024 next 389
2025-04-24 15:36:44.429585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b6000 of size 1024 next 390
2025-04-24 15:36:44.429594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b6400 of size 1024 next 391
2025-04-24 15:36:44.429603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b6800 of size 1024 next 392
2025-04-24 15:36:44.429611: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b6c00 of size 1024 next 393
2025-04-24 15:36:44.429620: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b7000 of size 4096 next 395
2025-04-24 15:36:44.429629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b8000 of size 4096 next 396
2025-04-24 15:36:44.429637: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86b9000 of size 4096 next 397
2025-04-24 15:36:44.429646: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ba000 of size 4096 next 398
2025-04-24 15:36:44.429655: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bb000 of size 4096 next 399
2025-04-24 15:36:44.429663: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bc000 of size 1024 next 400
2025-04-24 15:36:44.429672: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bc400 of size 1024 next 401
2025-04-24 15:36:44.429680: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bc800 of size 1024 next 402
2025-04-24 15:36:44.429689: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bcc00 of size 1024 next 403
2025-04-24 15:36:44.429698: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bd000 of size 1024 next 404
2025-04-24 15:36:44.429706: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bd400 of size 1024 next 407
2025-04-24 15:36:44.429715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bd800 of size 1024 next 408
2025-04-24 15:36:44.429724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bdc00 of size 1024 next 409
2025-04-24 15:36:44.429733: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86be000 of size 1024 next 410
2025-04-24 15:36:44.429741: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86be400 of size 1024 next 411
2025-04-24 15:36:44.429750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86be800 of size 4096 next 413
2025-04-24 15:36:44.429758: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86bf800 of size 4096 next 414
2025-04-24 15:36:44.429767: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c0800 of size 4096 next 415
2025-04-24 15:36:44.429775: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c1800 of size 4096 next 416
2025-04-24 15:36:44.429784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c2800 of size 4096 next 417
2025-04-24 15:36:44.429792: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c3800 of size 1024 next 418
2025-04-24 15:36:44.429802: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c3c00 of size 1024 next 419
2025-04-24 15:36:44.429810: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c4000 of size 1024 next 420
2025-04-24 15:36:44.429819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c4400 of size 1024 next 421
2025-04-24 15:36:44.429828: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c4800 of size 1024 next 422
2025-04-24 15:36:44.429836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c4c00 of size 1024 next 425
2025-04-24 15:36:44.429852: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c5000 of size 1024 next 426
2025-04-24 15:36:44.429861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c5400 of size 1024 next 427
2025-04-24 15:36:44.429869: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c5800 of size 1024 next 428
2025-04-24 15:36:44.429878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c5c00 of size 1024 next 429
2025-04-24 15:36:44.429886: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c6000 of size 4096 next 431
2025-04-24 15:36:44.429895: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c7000 of size 4096 next 432
2025-04-24 15:36:44.429904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c8000 of size 4096 next 433
2025-04-24 15:36:44.429912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86c9000 of size 4096 next 434
2025-04-24 15:36:44.429921: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86ca000 of size 4096 next 435
2025-04-24 15:36:44.429930: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86cb000 of size 1024 next 436
2025-04-24 15:36:44.429938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86cb400 of size 1024 next 437
2025-04-24 15:36:44.429947: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86cb800 of size 1024 next 438
2025-04-24 15:36:44.429955: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86cbc00 of size 1024 next 103
2025-04-24 15:36:44.429966: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b86cc000 of size 262144 next 102
2025-04-24 15:36:44.429975: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2b870c000 of size 999424 next 18446744073709551615
2025-04-24 15:36:44.429984: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304
2025-04-24 15:36:44.429993: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8800000 of size 589824 next 96
2025-04-24 15:36:44.430002: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8890000 of size 262144 next 117
2025-04-24 15:36:44.430010: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d0000 of size 1024 next 439
2025-04-24 15:36:44.430019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d0400 of size 1024 next 442
2025-04-24 15:36:44.430028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d0800 of size 1024 next 443
2025-04-24 15:36:44.430036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d0c00 of size 1024 next 444
2025-04-24 15:36:44.430045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d1000 of size 1024 next 445
2025-04-24 15:36:44.430054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d1400 of size 1024 next 446
2025-04-24 15:36:44.430062: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d1800 of size 4096 next 448
2025-04-24 15:36:44.430071: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d2800 of size 4096 next 449
2025-04-24 15:36:44.430079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d3800 of size 4096 next 450
2025-04-24 15:36:44.430088: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d4800 of size 4096 next 451
2025-04-24 15:36:44.430097: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d5800 of size 4096 next 452
2025-04-24 15:36:44.430105: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d6800 of size 1024 next 453
2025-04-24 15:36:44.430114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d6c00 of size 1024 next 454
2025-04-24 15:36:44.430122: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d7000 of size 1024 next 455
2025-04-24 15:36:44.430137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d7400 of size 1024 next 456
2025-04-24 15:36:44.430146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d7800 of size 1024 next 457
2025-04-24 15:36:44.430154: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d7c00 of size 1024 next 460
2025-04-24 15:36:44.430163: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d8000 of size 1024 next 461
2025-04-24 15:36:44.430172: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d8400 of size 1024 next 462
2025-04-24 15:36:44.430180: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d8800 of size 1024 next 463
2025-04-24 15:36:44.430189: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d8c00 of size 1024 next 464
2025-04-24 15:36:44.430197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88d9000 of size 4096 next 466
2025-04-24 15:36:44.430206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88da000 of size 4096 next 467
2025-04-24 15:36:44.430215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88db000 of size 4096 next 468
2025-04-24 15:36:44.430223: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88dc000 of size 4096 next 469
2025-04-24 15:36:44.430232: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88dd000 of size 4096 next 470
2025-04-24 15:36:44.430240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88de000 of size 1024 next 471
2025-04-24 15:36:44.430249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88de400 of size 1024 next 472
2025-04-24 15:36:44.430258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88de800 of size 1024 next 473
2025-04-24 15:36:44.430266: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88dec00 of size 1024 next 474
2025-04-24 15:36:44.430275: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88df000 of size 1024 next 475
2025-04-24 15:36:44.430283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88df400 of size 1024 next 478
2025-04-24 15:36:44.430292: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88df800 of size 1024 next 479
2025-04-24 15:36:44.430301: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88dfc00 of size 1024 next 480
2025-04-24 15:36:44.430310: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e0000 of size 1024 next 481
2025-04-24 15:36:44.430318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e0400 of size 1024 next 482
2025-04-24 15:36:44.430327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e0800 of size 4096 next 484
2025-04-24 15:36:44.430335: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e1800 of size 4096 next 485
2025-04-24 15:36:44.430344: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e2800 of size 4096 next 486
2025-04-24 15:36:44.430353: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e3800 of size 4096 next 487
2025-04-24 15:36:44.430361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e4800 of size 4096 next 488
2025-04-24 15:36:44.430380: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e5800 of size 1024 next 489
2025-04-24 15:36:44.430389: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e5c00 of size 1024 next 490
2025-04-24 15:36:44.430398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e6000 of size 1024 next 491
2025-04-24 15:36:44.430406: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e6400 of size 1024 next 492
2025-04-24 15:36:44.430415: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e6800 of size 1024 next 493
2025-04-24 15:36:44.430430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e6c00 of size 1024 next 496
2025-04-24 15:36:44.430439: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e7000 of size 1024 next 497
2025-04-24 15:36:44.430448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e7400 of size 1024 next 498
2025-04-24 15:36:44.430457: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e7800 of size 1024 next 499
2025-04-24 15:36:44.430465: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e7c00 of size 1024 next 500
2025-04-24 15:36:44.430474: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e8000 of size 4096 next 502
2025-04-24 15:36:44.430483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88e9000 of size 4096 next 503
2025-04-24 15:36:44.430491: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ea000 of size 4096 next 504
2025-04-24 15:36:44.430500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88eb000 of size 4096 next 505
2025-04-24 15:36:44.430508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ec000 of size 4096 next 506
2025-04-24 15:36:44.430517: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ed000 of size 1024 next 507
2025-04-24 15:36:44.430525: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ed400 of size 1024 next 508
2025-04-24 15:36:44.430534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ed800 of size 1024 next 509
2025-04-24 15:36:44.430543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88edc00 of size 1024 next 510
2025-04-24 15:36:44.430551: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ee000 of size 1024 next 511
2025-04-24 15:36:44.430560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ee400 of size 1024 next 514
2025-04-24 15:36:44.430568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ee800 of size 1024 next 515
2025-04-24 15:36:44.430577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88eec00 of size 1024 next 516
2025-04-24 15:36:44.430586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ef000 of size 1024 next 517
2025-04-24 15:36:44.430594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ef400 of size 1024 next 518
2025-04-24 15:36:44.430603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ef800 of size 4096 next 520
2025-04-24 15:36:44.430612: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f0800 of size 4096 next 521
2025-04-24 15:36:44.430621: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f1800 of size 4096 next 522
2025-04-24 15:36:44.430630: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f2800 of size 4096 next 523
2025-04-24 15:36:44.430638: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f3800 of size 4096 next 524
2025-04-24 15:36:44.430647: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f4800 of size 1024 next 525
2025-04-24 15:36:44.430656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f4c00 of size 1024 next 526
2025-04-24 15:36:44.430665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f5000 of size 1024 next 527
2025-04-24 15:36:44.430675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f5400 of size 1024 next 528
2025-04-24 15:36:44.430684: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f5800 of size 1024 next 529
2025-04-24 15:36:44.430694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f5c00 of size 1024 next 532
2025-04-24 15:36:44.430704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f6000 of size 1024 next 533
2025-04-24 15:36:44.430720: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f6400 of size 1024 next 534
2025-04-24 15:36:44.430728: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f6800 of size 1024 next 535
2025-04-24 15:36:44.430736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f6c00 of size 1024 next 536
2025-04-24 15:36:44.430745: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f7000 of size 4096 next 538
2025-04-24 15:36:44.430752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f8000 of size 4096 next 539
2025-04-24 15:36:44.430760: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88f9000 of size 4096 next 540
2025-04-24 15:36:44.430768: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fa000 of size 4096 next 541
2025-04-24 15:36:44.430776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fb000 of size 4096 next 542
2025-04-24 15:36:44.430784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fc000 of size 1024 next 543
2025-04-24 15:36:44.430792: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fc400 of size 1024 next 544
2025-04-24 15:36:44.430800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fc800 of size 1024 next 545
2025-04-24 15:36:44.430808: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fcc00 of size 1024 next 546
2025-04-24 15:36:44.430816: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fd000 of size 1024 next 547
2025-04-24 15:36:44.430824: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fd400 of size 1024 next 550
2025-04-24 15:36:44.430831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fd800 of size 1024 next 551
2025-04-24 15:36:44.430839: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fdc00 of size 1024 next 552
2025-04-24 15:36:44.430847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fe000 of size 1024 next 553
2025-04-24 15:36:44.430855: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fe400 of size 1024 next 554
2025-04-24 15:36:44.430862: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88fe800 of size 4096 next 556
2025-04-24 15:36:44.430870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b88ff800 of size 4096 next 557
2025-04-24 15:36:44.430878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8900800 of size 4096 next 558
2025-04-24 15:36:44.430886: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8901800 of size 4096 next 559
2025-04-24 15:36:44.430912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8902800 of size 4096 next 560
2025-04-24 15:36:44.430924: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8903800 of size 1024 next 561
2025-04-24 15:36:44.430932: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8903c00 of size 1024 next 562
2025-04-24 15:36:44.430941: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8904000 of size 1024 next 563
2025-04-24 15:36:44.430949: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8904400 of size 1024 next 564
2025-04-24 15:36:44.430958: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8904800 of size 1024 next 565
2025-04-24 15:36:44.430966: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8904c00 of size 1024 next 568
2025-04-24 15:36:44.430974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8905000 of size 1024 next 569
2025-04-24 15:36:44.430983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8905400 of size 1024 next 570
2025-04-24 15:36:44.430991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8905800 of size 1024 next 571
2025-04-24 15:36:44.430999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8905c00 of size 1024 next 572
2025-04-24 15:36:44.431015: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8906000 of size 4096 next 574
2025-04-24 15:36:44.431024: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8907000 of size 4096 next 575
2025-04-24 15:36:44.431032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8908000 of size 4096 next 576
2025-04-24 15:36:44.431040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8909000 of size 4096 next 577
2025-04-24 15:36:44.431049: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890a000 of size 4096 next 578
2025-04-24 15:36:44.431057: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890b000 of size 1024 next 579
2025-04-24 15:36:44.431065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890b400 of size 1024 next 580
2025-04-24 15:36:44.431074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890b800 of size 1024 next 581
2025-04-24 15:36:44.431082: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890bc00 of size 1024 next 582
2025-04-24 15:36:44.431090: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890c000 of size 1024 next 583
2025-04-24 15:36:44.431098: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890c400 of size 1024 next 586
2025-04-24 15:36:44.431106: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890c800 of size 1024 next 587
2025-04-24 15:36:44.431115: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890cc00 of size 1024 next 588
2025-04-24 15:36:44.431123: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890d000 of size 1024 next 589
2025-04-24 15:36:44.431131: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890d400 of size 1024 next 590
2025-04-24 15:36:44.431139: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890d800 of size 4096 next 592
2025-04-24 15:36:44.431148: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b890e800 of size 6144 next 112
2025-04-24 15:36:44.431157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8910000 of size 524288 next 111
2025-04-24 15:36:44.431165: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8990000 of size 262144 next 135
2025-04-24 15:36:44.431174: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b89d0000 of size 327680 next 124
2025-04-24 15:36:44.431182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8a20000 of size 589824 next 123
2025-04-24 15:36:44.431190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8ab0000 of size 262144 next 152
2025-04-24 15:36:44.431198: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8af0000 of size 327680 next 142
2025-04-24 15:36:44.431207: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8b40000 of size 786432 next 18446744073709551615
2025-04-24 15:36:44.431215: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608
2025-04-24 15:36:44.431224: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c00000 of size 4096 next 593
2025-04-24 15:36:44.431232: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c01000 of size 4096 next 594
2025-04-24 15:36:44.431240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c02000 of size 4096 next 595
2025-04-24 15:36:44.431249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c03000 of size 2048 next 598
2025-04-24 15:36:44.431257: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c03800 of size 2048 next 599
2025-04-24 15:36:44.431265: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c04000 of size 2048 next 600
2025-04-24 15:36:44.431273: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c04800 of size 2048 next 601
2025-04-24 15:36:44.431288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c05000 of size 2048 next 602
2025-04-24 15:36:44.431296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c05800 of size 256 next 605
2025-04-24 15:36:44.431305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c05900 of size 256 next 606
2025-04-24 15:36:44.431313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c05a00 of size 2048 next 604
2025-04-24 15:36:44.431321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c06200 of size 2048 next 609
2025-04-24 15:36:44.431329: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c06a00 of size 2048 next 610
2025-04-24 15:36:44.431338: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c07200 of size 2048 next 611
2025-04-24 15:36:44.431346: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c07a00 of size 2048 next 612
2025-04-24 15:36:44.431356: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c08200 of size 256 next 614
2025-04-24 15:36:44.431365: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c08300 of size 256 next 615
2025-04-24 15:36:44.431374: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c08400 of size 8192 next 613
2025-04-24 15:36:44.431382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c0a400 of size 8192 next 618
2025-04-24 15:36:44.431392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c0c400 of size 8192 next 619
2025-04-24 15:36:44.431403: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c0e400 of size 8192 next 620
2025-04-24 15:36:44.431430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c10400 of size 8192 next 621
2025-04-24 15:36:44.431441: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c12400 of size 256 next 622
2025-04-24 15:36:44.431449: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c12500 of size 256 next 623
2025-04-24 15:36:44.431459: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c12600 of size 8192 next 624
2025-04-24 15:36:44.431468: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c14600 of size 8192 next 626
2025-04-24 15:36:44.431476: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c16600 of size 8192 next 627
2025-04-24 15:36:44.431485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c18600 of size 8192 next 628
2025-04-24 15:36:44.431494: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c1a600 of size 8192 next 629
2025-04-24 15:36:44.431503: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c1c600 of size 2048 next 630
2025-04-24 15:36:44.431512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c1ce00 of size 2048 next 632
2025-04-24 15:36:44.431522: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c1d600 of size 2048 next 633
2025-04-24 15:36:44.431531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c1de00 of size 2048 next 634
2025-04-24 15:36:44.431542: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c1e600 of size 2048 next 635
2025-04-24 15:36:44.431551: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2b8c1ee00 of size 463360 next 160
2025-04-24 15:36:44.431560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8c90000 of size 589824 next 158
2025-04-24 15:36:44.431570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8d20000 of size 524288 next 171
2025-04-24 15:36:44.431580: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7fc2b8da0000 of size 1048576 next 222
2025-04-24 15:36:44.431589: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8ea0000 of size 1048576 next 190
2025-04-24 15:36:44.431608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b8fa0000 of size 1048576 next 189
2025-04-24 15:36:44.431618: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b90a0000 of size 1048576 next 203
2025-04-24 15:36:44.431628: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b91a0000 of size 1048576 next 251
2025-04-24 15:36:44.431638: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc2b92a0000 of size 1441792 next 18446744073709551615
2025-04-24 15:36:44.431647: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576
2025-04-24 15:36:44.431656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00000 of size 1280 next 1
2025-04-24 15:36:44.431665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00500 of size 256 next 5
2025-04-24 15:36:44.431675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00600 of size 256 next 8
2025-04-24 15:36:44.431685: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00700 of size 256 next 9
2025-04-24 15:36:44.431694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00800 of size 256 next 10
2025-04-24 15:36:44.431703: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00900 of size 256 next 11
2025-04-24 15:36:44.431712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00a00 of size 256 next 12
2025-04-24 15:36:44.431721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00b00 of size 256 next 13
2025-04-24 15:36:44.431730: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00c00 of size 256 next 17
2025-04-24 15:36:44.431739: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00d00 of size 256 next 19
2025-04-24 15:36:44.431748: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00e00 of size 256 next 20
2025-04-24 15:36:44.431757: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee00f00 of size 256 next 21
2025-04-24 15:36:44.431766: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01000 of size 256 next 22
2025-04-24 15:36:44.431776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01100 of size 256 next 24
2025-04-24 15:36:44.431785: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01200 of size 256 next 25
2025-04-24 15:36:44.431793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01300 of size 256 next 23
2025-04-24 15:36:44.431803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01400 of size 256 next 28
2025-04-24 15:36:44.431812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01500 of size 256 next 29
2025-04-24 15:36:44.431820: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01600 of size 256 next 30
2025-04-24 15:36:44.431830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01700 of size 256 next 31
2025-04-24 15:36:44.431839: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01800 of size 256 next 33
2025-04-24 15:36:44.431847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01900 of size 256 next 34
2025-04-24 15:36:44.431856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01a00 of size 1024 next 32
2025-04-24 15:36:44.431865: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee01e00 of size 1024 next 37
2025-04-24 15:36:44.431873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee02200 of size 1024 next 38
2025-04-24 15:36:44.431881: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee02600 of size 1024 next 39
2025-04-24 15:36:44.431889: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee02a00 of size 1024 next 40
2025-04-24 15:36:44.431898: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee02e00 of size 1024 next 41
2025-04-24 15:36:44.431913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee03200 of size 1024 next 43
2025-04-24 15:36:44.431921: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee03600 of size 1024 next 44
2025-04-24 15:36:44.431929: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee03a00 of size 1024 next 45
2025-04-24 15:36:44.431938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee03e00 of size 1024 next 46
2025-04-24 15:36:44.431946: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04200 of size 256 next 48
2025-04-24 15:36:44.431955: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04300 of size 256 next 49
2025-04-24 15:36:44.431963: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04400 of size 256 next 50
2025-04-24 15:36:44.431971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04500 of size 256 next 51
2025-04-24 15:36:44.431979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04600 of size 256 next 52
2025-04-24 15:36:44.431987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04700 of size 256 next 53
2025-04-24 15:36:44.431996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04800 of size 256 next 56
2025-04-24 15:36:44.432004: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04900 of size 256 next 57
2025-04-24 15:36:44.432012: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04a00 of size 256 next 58
2025-04-24 15:36:44.432020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04b00 of size 256 next 14
2025-04-24 15:36:44.432029: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04c00 of size 256 next 15
2025-04-24 15:36:44.432037: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04d00 of size 256 next 16
2025-04-24 15:36:44.432045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee04e00 of size 1024 next 60
2025-04-24 15:36:44.432053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee05200 of size 1024 next 61
2025-04-24 15:36:44.432062: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee05600 of size 1024 next 62
2025-04-24 15:36:44.432070: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee05a00 of size 1024 next 63
2025-04-24 15:36:44.432078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee05e00 of size 1024 next 64
2025-04-24 15:36:44.432086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06200 of size 256 next 66
2025-04-24 15:36:44.432095: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06300 of size 256 next 67
2025-04-24 15:36:44.432103: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06400 of size 256 next 68
2025-04-24 15:36:44.432111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06500 of size 256 next 69
2025-04-24 15:36:44.432119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06600 of size 256 next 70
2025-04-24 15:36:44.432128: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06700 of size 256 next 73
2025-04-24 15:36:44.432136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06800 of size 256 next 74
2025-04-24 15:36:44.432144: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06900 of size 256 next 75
2025-04-24 15:36:44.432153: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06a00 of size 256 next 76
2025-04-24 15:36:44.432161: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06b00 of size 256 next 77
2025-04-24 15:36:44.432169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee06c00 of size 1024 next 79
2025-04-24 15:36:44.432183: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee07000 of size 1024 next 80
2025-04-24 15:36:44.432192: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee07400 of size 1024 next 81
2025-04-24 15:36:44.432200: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee07800 of size 1024 next 82
2025-04-24 15:36:44.432208: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee07c00 of size 1024 next 83
2025-04-24 15:36:44.432216: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08000 of size 256 next 85
2025-04-24 15:36:44.432224: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08100 of size 256 next 86
2025-04-24 15:36:44.432233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08200 of size 512 next 84
2025-04-24 15:36:44.432241: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08400 of size 512 next 88
2025-04-24 15:36:44.432250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08600 of size 512 next 89
2025-04-24 15:36:44.432258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08800 of size 512 next 90
2025-04-24 15:36:44.432266: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08a00 of size 512 next 91
2025-04-24 15:36:44.432275: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08c00 of size 256 next 93
2025-04-24 15:36:44.432283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08d00 of size 256 next 94
2025-04-24 15:36:44.432291: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee08e00 of size 512 next 92
2025-04-24 15:36:44.432298: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee09000 of size 512 next 97
2025-04-24 15:36:44.432306: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee09200 of size 512 next 98
2025-04-24 15:36:44.432314: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee09400 of size 512 next 99
2025-04-24 15:36:44.432322: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee09600 of size 512 next 2
2025-04-24 15:36:44.432331: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee09800 of size 256 next 3
2025-04-24 15:36:44.432339: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee09900 of size 256 next 4
2025-04-24 15:36:44.432347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee09a00 of size 16384 next 18
2025-04-24 15:36:44.432355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee0da00 of size 256 next 100
2025-04-24 15:36:44.432364: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee0db00 of size 256 next 101
2025-04-24 15:36:44.432372: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee0dc00 of size 2048 next 104
2025-04-24 15:36:44.432380: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee0e400 of size 2048 next 105
2025-04-24 15:36:44.432388: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee0ec00 of size 2048 next 106
2025-04-24 15:36:44.432396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee0f400 of size 2048 next 107
2025-04-24 15:36:44.432405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee0fc00 of size 2048 next 108
2025-04-24 15:36:44.432413: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee10400 of size 256 next 109
2025-04-24 15:36:44.432421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee10500 of size 256 next 110
2025-04-24 15:36:44.432429: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee10600 of size 2048 next 113
2025-04-24 15:36:44.432437: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee10e00 of size 2048 next 114
2025-04-24 15:36:44.432446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee11600 of size 2048 next 115
2025-04-24 15:36:44.432460: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee11e00 of size 2048 next 116
2025-04-24 15:36:44.432469: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee12600 of size 2048 next 6
2025-04-24 15:36:44.432477: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee12e00 of size 37632 next 7
2025-04-24 15:36:44.432485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1c100 of size 512 next 118
2025-04-24 15:36:44.432494: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1c300 of size 512 next 119
2025-04-24 15:36:44.432502: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1c500 of size 512 next 120
2025-04-24 15:36:44.432510: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1c700 of size 512 next 121
2025-04-24 15:36:44.432518: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1c900 of size 512 next 122
2025-04-24 15:36:44.432537: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1cb00 of size 512 next 125
2025-04-24 15:36:44.432546: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1cd00 of size 512 next 126
2025-04-24 15:36:44.432555: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1cf00 of size 512 next 127
2025-04-24 15:36:44.432563: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1d100 of size 512 next 128
2025-04-24 15:36:44.432571: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1d300 of size 512 next 129
2025-04-24 15:36:44.432579: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1d500 of size 2048 next 130
2025-04-24 15:36:44.432588: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1dd00 of size 2048 next 131
2025-04-24 15:36:44.432596: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1e500 of size 2048 next 132
2025-04-24 15:36:44.432604: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1ed00 of size 2048 next 133
2025-04-24 15:36:44.432612: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1f500 of size 2048 next 134
2025-04-24 15:36:44.432620: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1fd00 of size 512 next 136
2025-04-24 15:36:44.432629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee1ff00 of size 512 next 137
2025-04-24 15:36:44.432637: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee20100 of size 512 next 138
2025-04-24 15:36:44.432645: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee20300 of size 512 next 139
2025-04-24 15:36:44.432653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee20500 of size 512 next 140
2025-04-24 15:36:44.432662: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee20700 of size 512 next 141
2025-04-24 15:36:44.432670: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee20900 of size 512 next 143
2025-04-24 15:36:44.432678: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee20b00 of size 512 next 144
2025-04-24 15:36:44.432686: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee20d00 of size 512 next 145
2025-04-24 15:36:44.432695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee20f00 of size 512 next 146
2025-04-24 15:36:44.432703: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee21100 of size 2048 next 147
2025-04-24 15:36:44.432711: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee21900 of size 2048 next 148
2025-04-24 15:36:44.432719: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee22100 of size 2048 next 149
2025-04-24 15:36:44.432728: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee22900 of size 2048 next 150
2025-04-24 15:36:44.432745: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee23100 of size 2048 next 151
2025-04-24 15:36:44.432754: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee23900 of size 512 next 153
2025-04-24 15:36:44.432763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee23b00 of size 512 next 154
2025-04-24 15:36:44.432771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee23d00 of size 512 next 155
2025-04-24 15:36:44.432779: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee23f00 of size 512 next 156
2025-04-24 15:36:44.432787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee24100 of size 512 next 157
2025-04-24 15:36:44.432796: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee24300 of size 512 next 161
2025-04-24 15:36:44.432804: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee24500 of size 512 next 162
2025-04-24 15:36:44.432813: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee24700 of size 512 next 163
2025-04-24 15:36:44.432821: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee24900 of size 512 next 164
2025-04-24 15:36:44.432829: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee24b00 of size 512 next 165
2025-04-24 15:36:44.432837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee24d00 of size 2048 next 166
2025-04-24 15:36:44.432846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee25500 of size 2048 next 167
2025-04-24 15:36:44.432854: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee25d00 of size 2048 next 168
2025-04-24 15:36:44.432862: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee26500 of size 2048 next 169
2025-04-24 15:36:44.432870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee26d00 of size 2048 next 170
2025-04-24 15:36:44.432879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee27500 of size 1024 next 172
2025-04-24 15:36:44.432887: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee27900 of size 1024 next 173
2025-04-24 15:36:44.432895: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee27d00 of size 1024 next 174
2025-04-24 15:36:44.432903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee28100 of size 1024 next 175
2025-04-24 15:36:44.432912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee28500 of size 1024 next 176
2025-04-24 15:36:44.432920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee28900 of size 256 next 178
2025-04-24 15:36:44.432928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee28a00 of size 256 next 179
2025-04-24 15:36:44.432936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee28b00 of size 1024 next 177
2025-04-24 15:36:44.432945: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee28f00 of size 1024 next 182
2025-04-24 15:36:44.432953: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee29300 of size 1024 next 183
2025-04-24 15:36:44.432961: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee29700 of size 1024 next 184
2025-04-24 15:36:44.432969: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee29b00 of size 1024 next 185
2025-04-24 15:36:44.432978: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee29f00 of size 256 next 187
2025-04-24 15:36:44.432986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee2a000 of size 256 next 188
2025-04-24 15:36:44.432994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee2a100 of size 4096 next 186
2025-04-24 15:36:44.433003: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee2b100 of size 4096 next 42
2025-04-24 15:36:44.433011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee2c100 of size 65536 next 36
2025-04-24 15:36:44.433026: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee3c100 of size 65536 next 35
2025-04-24 15:36:44.433034: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee4c100 of size 4096 next 235
2025-04-24 15:36:44.433042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee4d100 of size 4096 next 236
2025-04-24 15:36:44.433051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee4e100 of size 4096 next 237
2025-04-24 15:36:44.433059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee4f100 of size 4096 next 238
2025-04-24 15:36:44.433067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee50100 of size 1024 next 239
2025-04-24 15:36:44.433075: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee50500 of size 1024 next 240
2025-04-24 15:36:44.433084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee50900 of size 1024 next 241
2025-04-24 15:36:44.433092: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee50d00 of size 1024 next 242
2025-04-24 15:36:44.433100: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee51100 of size 1024 next 243
2025-04-24 15:36:44.433108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee51500 of size 1024 next 246
2025-04-24 15:36:44.433116: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee51900 of size 1024 next 247
2025-04-24 15:36:44.433125: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee51d00 of size 1024 next 248
2025-04-24 15:36:44.433133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee52100 of size 1024 next 249
2025-04-24 15:36:44.433141: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee52500 of size 1024 next 250
2025-04-24 15:36:44.433149: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee52900 of size 4096 next 252
2025-04-24 15:36:44.433158: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee53900 of size 4096 next 253
2025-04-24 15:36:44.433166: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee54900 of size 4096 next 254
2025-04-24 15:36:44.433174: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee55900 of size 4096 next 255
2025-04-24 15:36:44.433182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee56900 of size 4096 next 256
2025-04-24 15:36:44.433190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee57900 of size 1024 next 257
2025-04-24 15:36:44.433199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee57d00 of size 1024 next 258
2025-04-24 15:36:44.433207: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee58100 of size 1024 next 259
2025-04-24 15:36:44.433215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee58500 of size 1024 next 260
2025-04-24 15:36:44.433223: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee58900 of size 1024 next 261
2025-04-24 15:36:44.433231: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee58d00 of size 1024 next 264
2025-04-24 15:36:44.433240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee59100 of size 1024 next 265
2025-04-24 15:36:44.433248: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee59500 of size 1024 next 266
2025-04-24 15:36:44.433256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee59900 of size 1024 next 267
2025-04-24 15:36:44.433264: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee59d00 of size 1024 next 268
2025-04-24 15:36:44.433273: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5a100 of size 4096 next 270
2025-04-24 15:36:44.433281: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5b100 of size 4096 next 271
2025-04-24 15:36:44.433296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5c100 of size 4096 next 272
2025-04-24 15:36:44.433305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5d100 of size 4096 next 273
2025-04-24 15:36:44.433313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5e100 of size 4096 next 274
2025-04-24 15:36:44.433321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5f100 of size 1024 next 275
2025-04-24 15:36:44.433329: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5f500 of size 1024 next 276
2025-04-24 15:36:44.433337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5f900 of size 1024 next 277
2025-04-24 15:36:44.433345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee5fd00 of size 1024 next 278
2025-04-24 15:36:44.433353: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee60100 of size 1024 next 279
2025-04-24 15:36:44.433361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee60500 of size 1024 next 282
2025-04-24 15:36:44.433369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee60900 of size 1024 next 283
2025-04-24 15:36:44.433378: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee60d00 of size 1024 next 284
2025-04-24 15:36:44.433386: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee61100 of size 1024 next 285
2025-04-24 15:36:44.433394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee61500 of size 1024 next 286
2025-04-24 15:36:44.433403: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee61900 of size 4096 next 288
2025-04-24 15:36:44.433411: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee62900 of size 6144 next 27
2025-04-24 15:36:44.433419: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee64100 of size 147456 next 26
2025-04-24 15:36:44.433428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee88100 of size 65536 next 47
2025-04-24 15:36:44.433436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eee98100 of size 65536 next 59
2025-04-24 15:36:44.433444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eeea8100 of size 65536 next 65
2025-04-24 15:36:44.433453: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc3eeeb8100 of size 294656 next 18446744073709551615
2025-04-24 15:36:44.433461: I tensorflow/core/common_runtime/bfc_allocator.cc:995]      Summary of in-use Chunks by size: 
2025-04-24 15:36:44.433498: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 65 Chunks of size 256 totalling 16.2KiB
2025-04-24 15:36:44.433510: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB
2025-04-24 15:36:44.433519: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 250 Chunks of size 1024 totalling 250.0KiB
2025-04-24 15:36:44.433528: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB
2025-04-24 15:36:44.433538: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 2048 totalling 80.0KiB
2025-04-24 15:36:44.433547: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 117 Chunks of size 4096 totalling 468.0KiB
2025-04-24 15:36:44.433556: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 6144 totalling 12.0KiB
2025-04-24 15:36:44.433565: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB
2025-04-24 15:36:44.433574: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 10 Chunks of size 8192 totalling 80.0KiB
2025-04-24 15:36:44.433583: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB
2025-04-24 15:36:44.433592: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB
2025-04-24 15:36:44.433610: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB
2025-04-24 15:36:44.433620: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB
2025-04-24 15:36:44.433630: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB
2025-04-24 15:36:44.433639: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB
2025-04-24 15:36:44.433648: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB
2025-04-24 15:36:44.433658: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB
2025-04-24 15:36:44.433667: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB
2025-04-24 15:36:44.433676: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB
2025-04-24 15:36:44.433685: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB
2025-04-24 15:36:44.433694: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1048576 totalling 25.00MiB
2025-04-24 15:36:44.433703: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 18 Chunks of size 1310720 totalling 22.50MiB
2025-04-24 15:36:44.433712: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB
2025-04-24 15:36:44.433721: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB
2025-04-24 15:36:44.433730: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 2097152 totalling 4.00MiB
2025-04-24 15:36:44.433740: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 23 Chunks of size 2359296 totalling 51.75MiB
2025-04-24 15:36:44.433749: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 4194304 totalling 8.00MiB
2025-04-24 15:36:44.433758: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 9437184 totalling 9.00MiB
2025-04-24 15:36:44.433767: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 15597568 totalling 14.88MiB
2025-04-24 15:36:44.433776: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 145.48MiB
2025-04-24 15:36:44.433786: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 185466880 memory_limit_: 185466880 available bytes: 0 curr_region_allocation_bytes_: 268435456
2025-04-24 15:36:44.433799: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: 
Limit:                   185466880
InUse:                   152546816
MaxInUse:                165078528
NumAllocs:                    2110
MaxAllocSize:             15597568

2025-04-24 15:36:44.433824: W tensorflow/core/common_runtime/bfc_allocator.cc:439] __*****___******___******xxx********************************_**_**************************_*********
2025-04-24 15:36:44.433864: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at random_op.cc:77 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


 Exception in mask_detect :  OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform]
we want to redo the detection
Using TensorFlow backend.
max_time_sub_proc : 3600
erreur pendant la detection
Useless call to update_current_state
in case -12
caffe_path_current : 
About to save ! 1
Inside saveOutput : final : True verbose : False
eke 12-6-18 : saveMask need to be cleaned for new output !
ERROR : mask output needs to be a dictionnary now !
No output to save, continue without doing anything !

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : -12
ERROR : 'int' object is not subscriptable
reconnect to base !
warning , we can't find thcl infos in json_data
warning , we can't find pdt infos in json_data


#&_# TEST FAILED #&_# : tests/mask_test #&_# 


Error : invalid literal for int() with base 10: "'int' object is not subscriptable"
/home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_3ffe22aa43084fd48098e0c6d2c2993c4434ab23
SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_3ffe22aa43084fd48098e0c6d2c2993c4434ab23','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/April/24042025/python_test3//data_2/data_log/job/2025/April/24042025/python_test3/log-python3----short_python3--v--marlene-15:35:01.txt','mask_detection','unknown');

#&_# END OF TEST #&_# : tests/mask_test #&_# 





#&_# BEGIN OF TEST : tests/datou_test #&_# 


/home/admin/workarea/git/Velours/python/tests/datou_test.py
<module 'tests.datou_test' from '/home/admin/workarea/git/Velours/python/tests/datou_test.py'>
Datou All Test
python version used : 3

############################### TEST sam  ################################
TEST SAM
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4573
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4573
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4573
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : sam
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (1189321094) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 1189321094
download finish for photo 1189321094
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.18994355201721191
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
WARNING : we have an input that is not a photo, we should get rid of it
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:sam
Thu Apr 24 15:36:45 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1745501804_1172007_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094}
map_photo_id_path_extension : {1189321094: {'path': 'temp/1745501804_1172007_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}}
map_subphoto_mainphoto : {}

Beginning of datou step sam !
pht : 4677
Inside sam : nb paths : 1
ERROR in datou_step_exec, will save and exit !
CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec
    output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2430, in datou_step_exec
    return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 367, in datou_step_sam
    sam.to(device=device)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to
    return self._apply(convert)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 602, in _apply
    param_applied = fn(param)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

[1189321094]
 map_info['map_portfolio_photo'] : {}
 final :  True mtd_id 4573 list_pids : [1189321094]


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 1
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('4573', None, '1189321094', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 100 in test_sam>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 819 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2346 in datou_exec>, 'CUDA error: out of memory\\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.']", '-1', '-1.0', '501120777', '1.0', None)]
time used for this insertion : 0.012097358703613281
 save_final 
ERROR in last step sam, CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

time spend for datou_step_exec : 6.016534090042114
time spend to save output : 0.019465923309326172
total time spend for step 0 : 6.03600001335144
need to delete datou_research and reload, so keep current state 1
need to delete datou_research and reload, so keep current state 1
caffe_path_current : 
About to save ! 2
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : None
ERROR
nb objects espect :  98
nb_objects detect : 0
ERROR sam FAILED

############################### TEST frcnn  ################################
test frcnn
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4184
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4184
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4184
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : frcnn
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (917754606) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 917754606
download finish for photo 917754606
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.1432788372039795
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:frcnn
Thu Apr 24 15:36:51 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1745501811_1172007_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606}
map_photo_id_path_extension : {917754606: {'path': 'temp/1745501811_1172007_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou step Faster rcnn !
classes : ['background', 'plaque']
pht : 4370
caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'}
To loadFromThcl()
model_param file didn't exist 
model_name : detection_plaque_valcor_010622
model_type : caffe_faster_rcnn
list file need :
['caffemodel', 'test.prototxt']
file exist in s3 :
['caffemodel', 'test.prototxt']
file manque in s3 : 
[]
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0424 15:36:52.590632 1172007 common.cpp:114] Cannot create Cublas handle. Cublas won't be available.
E0424 15:36:52.595288 1172007 common.cpp:121] Cannot create Curand generator. Curand won't be available.
F0424 15:36:52.622824 1172007 syncedmem.hpp:22] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
Command terminated by signal 6
14.24user 9.85system 1:26.90elapsed 27%CPU (0avgtext+0avgdata 2311580maxresident)k
2182808inputs+3448outputs (4809major+1128089minor)pagefaults 0swaps