python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120
 import MySQLdb succeeded 
Import error (python version)
python version = 3
warning , we can't find thcl infos in json_data
warning , we can't find pdt infos in json_data
 list_job_run_as_list :  ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel']
python version used : 3
 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')]


#&_# BEGIN OF TEST : tests/mask_test #&_# 


/home/admin/workarea/git/Velours/python/tests/mask_test.py
<module 'tests.mask_test' from '/home/admin/workarea/git/Velours/python/tests/mask_test.py'>
Test mask-detection 
python version used : 3

############################### TEST memory used  ################################
free memory at begining : 
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 358

run mask_detect

Inside batchDatouExec : verbose : False
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
List Step Type Loaded in datou : mask_detect
list_input_json : []
origin
BFwe have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.14841890335083008
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Wed May 14 06:35:26 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3000
l 3632 free memory gpu now : 358
wait 20 seconds
l 3637 free memory gpu now : 358  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
/home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
2025-05-14 06:35:49.566870: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-05-14 06:35:49.599258: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz
2025-05-14 06:35:49.601452: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f2c90000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-05-14 06:35:49.601505: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-05-14 06:35:49.605685: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-05-14 06:35:49.853270: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x20180fc0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-05-14 06:35:49.853330: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-05-14 06:35:49.854359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-05-14 06:35:49.854829: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-05-14 06:35:49.858182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-05-14 06:35:49.861359: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-05-14 06:35:49.861910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-05-14 06:35:49.865151: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-05-14 06:35:49.866256: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-05-14 06:35:49.870974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-05-14 06:35:49.871917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-05-14 06:35:49.872028: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-05-14 06:35:49.872512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-05-14 06:35:49.872528: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-05-14 06:35:49.872537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-05-14 06:35:49.873249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 206 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
To do loadFromThcl(), then load ParamDescType : thcl454
 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3473
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
2025-05-14 06:35:50.442871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-05-14 06:35:50.443005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-05-14 06:35:50.443042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-05-14 06:35:50.443071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-05-14 06:35:50.443100: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-05-14 06:35:50.443128: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-05-14 06:35:50.443171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-05-14 06:35:50.443201: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-05-14 06:35:50.444246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-05-14 06:35:50.445020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-05-14 06:35:50.445048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-05-14 06:35:50.445064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-05-14 06:35:50.445079: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-05-14 06:35:50.445093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-05-14 06:35:50.445107: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-05-14 06:35:50.445121: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-05-14 06:35:50.445136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-05-14 06:35:50.445797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-05-14 06:35:50.445824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-05-14 06:35:50.445832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-05-14 06:35:50.445840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-05-14 06:35:50.446567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 206 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
2025-05-14 06:35:51.250796: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:35:51.251373: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 28.80M (30199040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:35:51.251867: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 25.92M (27179264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:35:51.252337: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 23.33M (24461568 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:35:51.252795: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 21.00M (22015488 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:35:51.393205: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:35:51.393712: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:01.394755: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:01.395567: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:01.395618: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25MiB (rounded to 2359296)
Current allocation summary follows.
2025-05-14 06:36:01.395648: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc
2025-05-14 06:36:01.395666: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): 	Total Chunks: 59, Chunks in use: 59. 14.8KiB allocated for chunks. 14.8KiB in use in bin. 8.8KiB client-requested in use in bin.
2025-05-14 06:36:01.395680: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): 	Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin.
2025-05-14 06:36:01.395694: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): 	Total Chunks: 96, Chunks in use: 96. 96.2KiB allocated for chunks. 96.2KiB in use in bin. 96.0KiB client-requested in use in bin.
2025-05-14 06:36:01.395707: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): 	Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin.
2025-05-14 06:36:01.395720: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): 	Total Chunks: 40, Chunks in use: 40. 165.5KiB allocated for chunks. 165.5KiB in use in bin. 160.0KiB client-requested in use in bin.
2025-05-14 06:36:01.395732: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:01.395746: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): 	Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2025-05-14 06:36:01.395759: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): 	Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin.
2025-05-14 06:36:01.395772: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): 	Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin.
2025-05-14 06:36:01.395785: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): 	Total Chunks: 5, Chunks in use: 4. 769.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin.
2025-05-14 06:36:01.395797: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): 	Total Chunks: 8, Chunks in use: 7. 2.16MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin.
2025-05-14 06:36:01.395810: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): 	Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin.
2025-05-14 06:36:01.395822: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): 	Total Chunks: 16, Chunks in use: 14. 17.62MiB allocated for chunks. 15.38MiB in use in bin. 14.00MiB client-requested in use in bin.
2025-05-14 06:36:01.395835: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): 	Total Chunks: 10, Chunks in use: 10. 23.65MiB allocated for chunks. 23.65MiB in use in bin. 22.25MiB client-requested in use in bin.
2025-05-14 06:36:01.395846: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:01.395857: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:01.395868: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:01.395887: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:01.395898: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:01.395910: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:01.395922: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:01.395934: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.25MiB was 2.00MiB, Chunk State: 
2025-05-14 06:36:01.395944: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 19814144
2025-05-14 06:36:01.395960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0000000 of size 2359296 next 245
2025-05-14 06:36:01.395970: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0240000 of size 1048576 next 269
2025-05-14 06:36:01.395980: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab0340000 of size 1310720 next 263
2025-05-14 06:36:01.395990: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0480000 of size 2359296 next 262
2025-05-14 06:36:01.395999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab06c0000 of size 1048576 next 287
2025-05-14 06:36:01.396009: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab07c0000 of size 1310720 next 281
2025-05-14 06:36:01.396019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0900000 of size 2359296 next 280
2025-05-14 06:36:01.396028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0b40000 of size 1048576 next 304
2025-05-14 06:36:01.396037: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0c40000 of size 1310720 next 298
2025-05-14 06:36:01.396046: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0d80000 of size 2359296 next 297
2025-05-14 06:36:01.396056: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0fc0000 of size 3299072 next 18446744073709551615
2025-05-14 06:36:01.396066: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216
2025-05-14 06:36:01.396076: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2000000 of size 2359296 next 181
2025-05-14 06:36:01.396085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2240000 of size 2097152 next 196
2025-05-14 06:36:01.396095: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2440000 of size 1048576 next 216
2025-05-14 06:36:01.396104: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2540000 of size 1310720 next 209
2025-05-14 06:36:01.396114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2680000 of size 2359296 next 208
2025-05-14 06:36:01.396123: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab28c0000 of size 1048576 next 234
2025-05-14 06:36:01.396132: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab29c0000 of size 1310720 next 228
2025-05-14 06:36:01.396141: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2b00000 of size 2359296 next 227
2025-05-14 06:36:01.396151: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2d40000 of size 2883584 next 18446744073709551615
2025-05-14 06:36:01.396160: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152
2025-05-14 06:36:01.396171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4600000 of size 147456 next 55
2025-05-14 06:36:01.396189: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4624000 of size 65536 next 78
2025-05-14 06:36:01.396199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4634000 of size 4096 next 191
2025-05-14 06:36:01.396208: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4635000 of size 4096 next 192
2025-05-14 06:36:01.396218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4636000 of size 4096 next 193
2025-05-14 06:36:01.396228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4637000 of size 256 next 194
2025-05-14 06:36:01.396237: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4637100 of size 256 next 195
2025-05-14 06:36:01.396247: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4637200 of size 4096 next 197
2025-05-14 06:36:01.396256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4638200 of size 4096 next 198
2025-05-14 06:36:01.396266: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4639200 of size 4096 next 199
2025-05-14 06:36:01.396275: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463a200 of size 4096 next 200
2025-05-14 06:36:01.396284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463b200 of size 4096 next 201
2025-05-14 06:36:01.396294: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463c200 of size 1024 next 202
2025-05-14 06:36:01.396303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463c600 of size 1024 next 204
2025-05-14 06:36:01.396312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463ca00 of size 1024 next 205
2025-05-14 06:36:01.396321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463ce00 of size 1024 next 206
2025-05-14 06:36:01.396331: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463d200 of size 1024 next 207
2025-05-14 06:36:01.396340: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463d600 of size 1024 next 210
2025-05-14 06:36:01.396350: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463da00 of size 1024 next 211
2025-05-14 06:36:01.396359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463de00 of size 1024 next 212
2025-05-14 06:36:01.396368: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463e200 of size 1024 next 213
2025-05-14 06:36:01.396378: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463e600 of size 1024 next 214
2025-05-14 06:36:01.396387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463ea00 of size 4096 next 215
2025-05-14 06:36:01.396397: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463fa00 of size 4096 next 217
2025-05-14 06:36:01.396406: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4640a00 of size 4096 next 218
2025-05-14 06:36:01.396416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4641a00 of size 4096 next 219
2025-05-14 06:36:01.396425: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4642a00 of size 4096 next 220
2025-05-14 06:36:01.396435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4643a00 of size 1024 next 221
2025-05-14 06:36:01.396444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4643e00 of size 1024 next 223
2025-05-14 06:36:01.396453: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4644200 of size 1024 next 224
2025-05-14 06:36:01.396462: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4644600 of size 1024 next 225
2025-05-14 06:36:01.396472: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4644a00 of size 1024 next 226
2025-05-14 06:36:01.396481: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4644e00 of size 1024 next 229
2025-05-14 06:36:01.396498: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4645200 of size 1024 next 230
2025-05-14 06:36:01.396508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4645600 of size 1024 next 231
2025-05-14 06:36:01.396517: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4645a00 of size 1024 next 232
2025-05-14 06:36:01.396527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4645e00 of size 1024 next 233
2025-05-14 06:36:01.396537: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4646200 of size 7680 next 72
2025-05-14 06:36:01.396546: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4648000 of size 147456 next 71
2025-05-14 06:36:01.396556: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab466c000 of size 131072 next 87
2025-05-14 06:36:01.396566: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468c000 of size 4096 next 289
2025-05-14 06:36:01.396575: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468d000 of size 4096 next 290
2025-05-14 06:36:01.396584: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468e000 of size 4096 next 291
2025-05-14 06:36:01.396594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468f000 of size 1024 next 292
2025-05-14 06:36:01.396603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468f400 of size 1024 next 293
2025-05-14 06:36:01.396612: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468f800 of size 1024 next 294
2025-05-14 06:36:01.396621: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468fc00 of size 1024 next 295
2025-05-14 06:36:01.396630: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4690000 of size 1024 next 296
2025-05-14 06:36:01.396640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4690400 of size 1024 next 299
2025-05-14 06:36:01.396649: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4690800 of size 1024 next 300
2025-05-14 06:36:01.396658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4690c00 of size 1024 next 301
2025-05-14 06:36:01.396667: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4691000 of size 1024 next 302
2025-05-14 06:36:01.396677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4691400 of size 1024 next 303
2025-05-14 06:36:01.396686: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4691800 of size 4096 next 305
2025-05-14 06:36:01.396696: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4692800 of size 4096 next 306
2025-05-14 06:36:01.396705: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4693800 of size 4096 next 307
2025-05-14 06:36:01.396715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4694800 of size 4096 next 308
2025-05-14 06:36:01.396724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4695800 of size 4096 next 309
2025-05-14 06:36:01.396734: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4696800 of size 1024 next 310
2025-05-14 06:36:01.396743: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4696c00 of size 1024 next 311
2025-05-14 06:36:01.396752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4697000 of size 1024 next 312
2025-05-14 06:36:01.396761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4697400 of size 1024 next 313
2025-05-14 06:36:01.396770: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4697800 of size 1024 next 314
2025-05-14 06:36:01.396779: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab4697c00 of size 214016 next 103
2025-05-14 06:36:01.396789: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab46cc000 of size 262144 next 102
2025-05-14 06:36:01.396799: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab470c000 of size 999424 next 18446744073709551615
2025-05-14 06:36:01.396816: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304
2025-05-14 06:36:01.396826: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4800000 of size 589824 next 96
2025-05-14 06:36:01.396836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4890000 of size 262144 next 117
2025-05-14 06:36:01.396846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab48d0000 of size 262144 next 112
2025-05-14 06:36:01.396856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4910000 of size 524288 next 111
2025-05-14 06:36:01.396865: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4990000 of size 262144 next 135
2025-05-14 06:36:01.396875: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab49d0000 of size 327680 next 124
2025-05-14 06:36:01.396884: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4a20000 of size 589824 next 123
2025-05-14 06:36:01.396894: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4ab0000 of size 262144 next 152
2025-05-14 06:36:01.396903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4af0000 of size 327680 next 142
2025-05-14 06:36:01.396913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4b40000 of size 786432 next 18446744073709551615
2025-05-14 06:36:01.396922: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608
2025-05-14 06:36:01.396931: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab4c00000 of size 589824 next 160
2025-05-14 06:36:01.396941: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4c90000 of size 589824 next 158
2025-05-14 06:36:01.396950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4d20000 of size 524288 next 171
2025-05-14 06:36:01.396960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab4da0000 of size 1048576 next 222
2025-05-14 06:36:01.396969: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4ea0000 of size 1048576 next 190
2025-05-14 06:36:01.396978: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4fa0000 of size 1048576 next 189
2025-05-14 06:36:01.396987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab50a0000 of size 1048576 next 203
2025-05-14 06:36:01.396997: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab51a0000 of size 1048576 next 251
2025-05-14 06:36:01.397007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab52a0000 of size 1441792 next 18446744073709551615
2025-05-14 06:36:01.397016: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576
2025-05-14 06:36:01.397026: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00000 of size 1280 next 1
2025-05-14 06:36:01.397036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00500 of size 256 next 5
2025-05-14 06:36:01.397045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00600 of size 256 next 8
2025-05-14 06:36:01.397054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00700 of size 256 next 9
2025-05-14 06:36:01.397064: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00800 of size 256 next 10
2025-05-14 06:36:01.397073: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00900 of size 256 next 11
2025-05-14 06:36:01.397082: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00a00 of size 256 next 12
2025-05-14 06:36:01.397091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00b00 of size 256 next 13
2025-05-14 06:36:01.397100: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00c00 of size 256 next 17
2025-05-14 06:36:01.397110: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00d00 of size 256 next 19
2025-05-14 06:36:01.397127: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00e00 of size 256 next 20
2025-05-14 06:36:01.397137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00f00 of size 256 next 21
2025-05-14 06:36:01.397146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01000 of size 256 next 22
2025-05-14 06:36:01.397156: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01100 of size 256 next 24
2025-05-14 06:36:01.397166: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01200 of size 256 next 25
2025-05-14 06:36:01.397175: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01300 of size 256 next 23
2025-05-14 06:36:01.397185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01400 of size 256 next 28
2025-05-14 06:36:01.397194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01500 of size 256 next 29
2025-05-14 06:36:01.397203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01600 of size 256 next 30
2025-05-14 06:36:01.397213: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01700 of size 256 next 31
2025-05-14 06:36:01.397222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01800 of size 256 next 33
2025-05-14 06:36:01.397231: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01900 of size 256 next 34
2025-05-14 06:36:01.397240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01a00 of size 1024 next 32
2025-05-14 06:36:01.397249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01e00 of size 1024 next 37
2025-05-14 06:36:01.397259: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee02200 of size 1024 next 38
2025-05-14 06:36:01.397268: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee02600 of size 1024 next 39
2025-05-14 06:36:01.397277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee02a00 of size 1024 next 40
2025-05-14 06:36:01.397287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee02e00 of size 1024 next 41
2025-05-14 06:36:01.397296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee03200 of size 1024 next 43
2025-05-14 06:36:01.397305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee03600 of size 1024 next 44
2025-05-14 06:36:01.397315: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee03a00 of size 1024 next 45
2025-05-14 06:36:01.397326: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee03e00 of size 1024 next 46
2025-05-14 06:36:01.397335: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04200 of size 256 next 48
2025-05-14 06:36:01.397344: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04300 of size 256 next 49
2025-05-14 06:36:01.397354: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04400 of size 256 next 50
2025-05-14 06:36:01.397363: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04500 of size 256 next 51
2025-05-14 06:36:01.397372: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04600 of size 256 next 52
2025-05-14 06:36:01.397381: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04700 of size 256 next 53
2025-05-14 06:36:01.397390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04800 of size 256 next 56
2025-05-14 06:36:01.397399: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04900 of size 256 next 57
2025-05-14 06:36:01.397408: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04a00 of size 256 next 58
2025-05-14 06:36:01.397418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04b00 of size 256 next 14
2025-05-14 06:36:01.397436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04c00 of size 256 next 15
2025-05-14 06:36:01.397446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04d00 of size 256 next 16
2025-05-14 06:36:01.397455: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04e00 of size 1024 next 60
2025-05-14 06:36:01.397464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee05200 of size 1024 next 61
2025-05-14 06:36:01.397473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee05600 of size 1024 next 62
2025-05-14 06:36:01.397483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee05a00 of size 1024 next 63
2025-05-14 06:36:01.397493: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee05e00 of size 1024 next 64
2025-05-14 06:36:01.397502: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06200 of size 256 next 66
2025-05-14 06:36:01.397511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06300 of size 256 next 67
2025-05-14 06:36:01.397521: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06400 of size 256 next 68
2025-05-14 06:36:01.397530: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06500 of size 256 next 69
2025-05-14 06:36:01.397539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06600 of size 256 next 70
2025-05-14 06:36:01.397548: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06700 of size 256 next 73
2025-05-14 06:36:01.397557: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06800 of size 256 next 74
2025-05-14 06:36:01.397566: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06900 of size 256 next 75
2025-05-14 06:36:01.397575: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06a00 of size 256 next 76
2025-05-14 06:36:01.397585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06b00 of size 256 next 77
2025-05-14 06:36:01.397594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06c00 of size 1024 next 79
2025-05-14 06:36:01.397603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee07000 of size 1024 next 80
2025-05-14 06:36:01.397612: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee07400 of size 1024 next 81
2025-05-14 06:36:01.397621: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee07800 of size 1024 next 82
2025-05-14 06:36:01.397631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee07c00 of size 1024 next 83
2025-05-14 06:36:01.397641: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08000 of size 256 next 85
2025-05-14 06:36:01.397650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08100 of size 256 next 86
2025-05-14 06:36:01.397660: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08200 of size 512 next 84
2025-05-14 06:36:01.397669: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08400 of size 512 next 88
2025-05-14 06:36:01.397679: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08600 of size 512 next 89
2025-05-14 06:36:01.397688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08800 of size 512 next 90
2025-05-14 06:36:01.397697: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08a00 of size 512 next 91
2025-05-14 06:36:01.397706: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08c00 of size 256 next 93
2025-05-14 06:36:01.397715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08d00 of size 256 next 94
2025-05-14 06:36:01.397724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08e00 of size 512 next 92
2025-05-14 06:36:01.397734: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09000 of size 512 next 97
2025-05-14 06:36:01.397752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09200 of size 512 next 98
2025-05-14 06:36:01.397762: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09400 of size 512 next 99
2025-05-14 06:36:01.397771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09600 of size 512 next 2
2025-05-14 06:36:01.397780: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09800 of size 256 next 3
2025-05-14 06:36:01.397790: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09900 of size 256 next 4
2025-05-14 06:36:01.397800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09a00 of size 16384 next 18
2025-05-14 06:36:01.397809: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0da00 of size 256 next 100
2025-05-14 06:36:01.397818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0db00 of size 256 next 101
2025-05-14 06:36:01.397828: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0dc00 of size 2048 next 104
2025-05-14 06:36:01.397838: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0e400 of size 2048 next 105
2025-05-14 06:36:01.397847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0ec00 of size 2048 next 106
2025-05-14 06:36:01.397856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0f400 of size 2048 next 107
2025-05-14 06:36:01.397865: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0fc00 of size 2048 next 108
2025-05-14 06:36:01.397874: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee10400 of size 256 next 109
2025-05-14 06:36:01.397883: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee10500 of size 256 next 110
2025-05-14 06:36:01.397893: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee10600 of size 2048 next 113
2025-05-14 06:36:01.397902: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee10e00 of size 2048 next 114
2025-05-14 06:36:01.397911: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee11600 of size 2048 next 115
2025-05-14 06:36:01.397920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee11e00 of size 2048 next 116
2025-05-14 06:36:01.397929: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee12600 of size 2048 next 6
2025-05-14 06:36:01.397938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee12e00 of size 37632 next 7
2025-05-14 06:36:01.397948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c100 of size 512 next 118
2025-05-14 06:36:01.397958: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c300 of size 512 next 119
2025-05-14 06:36:01.397968: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c500 of size 512 next 120
2025-05-14 06:36:01.397977: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c700 of size 512 next 121
2025-05-14 06:36:01.397986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c900 of size 512 next 122
2025-05-14 06:36:01.397995: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1cb00 of size 512 next 125
2025-05-14 06:36:01.398004: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1cd00 of size 512 next 126
2025-05-14 06:36:01.398013: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1cf00 of size 512 next 127
2025-05-14 06:36:01.398022: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1d100 of size 512 next 128
2025-05-14 06:36:01.398031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1d300 of size 512 next 129
2025-05-14 06:36:01.398041: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1d500 of size 2048 next 130
2025-05-14 06:36:01.398051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1dd00 of size 2048 next 131
2025-05-14 06:36:01.398068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1e500 of size 2048 next 132
2025-05-14 06:36:01.398078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1ed00 of size 2048 next 133
2025-05-14 06:36:01.398087: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1f500 of size 2048 next 134
2025-05-14 06:36:01.398096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1fd00 of size 512 next 136
2025-05-14 06:36:01.398106: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1ff00 of size 512 next 137
2025-05-14 06:36:01.398115: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20100 of size 512 next 138
2025-05-14 06:36:01.398124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20300 of size 512 next 139
2025-05-14 06:36:01.398133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20500 of size 512 next 140
2025-05-14 06:36:01.398143: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20700 of size 512 next 141
2025-05-14 06:36:01.398152: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20900 of size 512 next 143
2025-05-14 06:36:01.398161: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20b00 of size 512 next 144
2025-05-14 06:36:01.398170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20d00 of size 512 next 145
2025-05-14 06:36:01.398179: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20f00 of size 512 next 146
2025-05-14 06:36:01.398188: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee21100 of size 2048 next 147
2025-05-14 06:36:01.398198: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee21900 of size 2048 next 148
2025-05-14 06:36:01.398207: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee22100 of size 2048 next 149
2025-05-14 06:36:01.398217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee22900 of size 2048 next 150
2025-05-14 06:36:01.398226: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23100 of size 2048 next 151
2025-05-14 06:36:01.398235: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23900 of size 512 next 153
2025-05-14 06:36:01.398244: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23b00 of size 512 next 154
2025-05-14 06:36:01.398254: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23d00 of size 512 next 155
2025-05-14 06:36:01.398263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23f00 of size 512 next 156
2025-05-14 06:36:01.398272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24100 of size 512 next 157
2025-05-14 06:36:01.398282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24300 of size 512 next 161
2025-05-14 06:36:01.398291: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24500 of size 512 next 162
2025-05-14 06:36:01.398300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24700 of size 512 next 163
2025-05-14 06:36:01.398309: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24900 of size 512 next 164
2025-05-14 06:36:01.398318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24b00 of size 512 next 165
2025-05-14 06:36:01.398327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24d00 of size 2048 next 166
2025-05-14 06:36:01.398337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee25500 of size 2048 next 167
2025-05-14 06:36:01.398346: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee25d00 of size 2048 next 168
2025-05-14 06:36:01.398355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee26500 of size 2048 next 169
2025-05-14 06:36:01.398373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee26d00 of size 2048 next 170
2025-05-14 06:36:01.398383: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee27500 of size 1024 next 172
2025-05-14 06:36:01.398392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee27900 of size 1024 next 173
2025-05-14 06:36:01.398402: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee27d00 of size 1024 next 174
2025-05-14 06:36:01.398411: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28100 of size 1024 next 175
2025-05-14 06:36:01.398421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28500 of size 1024 next 176
2025-05-14 06:36:01.398430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28900 of size 256 next 178
2025-05-14 06:36:01.398439: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28a00 of size 256 next 179
2025-05-14 06:36:01.398448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28b00 of size 1024 next 177
2025-05-14 06:36:01.398458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28f00 of size 1024 next 182
2025-05-14 06:36:01.398467: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee29300 of size 1024 next 183
2025-05-14 06:36:01.398476: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee29700 of size 1024 next 184
2025-05-14 06:36:01.398485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee29b00 of size 1024 next 185
2025-05-14 06:36:01.398494: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee29f00 of size 256 next 187
2025-05-14 06:36:01.398503: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee2a000 of size 256 next 188
2025-05-14 06:36:01.398512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee2a100 of size 4096 next 186
2025-05-14 06:36:01.398522: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee2b100 of size 4096 next 42
2025-05-14 06:36:01.398531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee2c100 of size 65536 next 36
2025-05-14 06:36:01.398540: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee3c100 of size 65536 next 35
2025-05-14 06:36:01.398549: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee4c100 of size 4096 next 235
2025-05-14 06:36:01.398558: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee4d100 of size 4096 next 236
2025-05-14 06:36:01.398568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee4e100 of size 4096 next 237
2025-05-14 06:36:01.398577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee4f100 of size 4096 next 238
2025-05-14 06:36:01.398586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee50100 of size 1024 next 239
2025-05-14 06:36:01.398596: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee50500 of size 1024 next 240
2025-05-14 06:36:01.398605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee50900 of size 1024 next 241
2025-05-14 06:36:01.398614: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee50d00 of size 1024 next 242
2025-05-14 06:36:01.398624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee51100 of size 1024 next 243
2025-05-14 06:36:01.398633: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee51500 of size 1024 next 246
2025-05-14 06:36:01.398643: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee51900 of size 1024 next 247
2025-05-14 06:36:01.398651: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee51d00 of size 1024 next 248
2025-05-14 06:36:01.398661: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee52100 of size 1024 next 249
2025-05-14 06:36:01.398678: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee52500 of size 1024 next 250
2025-05-14 06:36:01.398688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee52900 of size 4096 next 252
2025-05-14 06:36:01.398698: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee53900 of size 4096 next 253
2025-05-14 06:36:01.398707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee54900 of size 4096 next 254
2025-05-14 06:36:01.398716: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee55900 of size 4096 next 255
2025-05-14 06:36:01.398725: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee56900 of size 4096 next 256
2025-05-14 06:36:01.398735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee57900 of size 1024 next 257
2025-05-14 06:36:01.398744: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee57d00 of size 1024 next 258
2025-05-14 06:36:01.398753: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee58100 of size 1024 next 259
2025-05-14 06:36:01.398763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee58500 of size 1024 next 260
2025-05-14 06:36:01.398772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee58900 of size 1024 next 261
2025-05-14 06:36:01.398781: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee58d00 of size 1024 next 264
2025-05-14 06:36:01.398790: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee59100 of size 1024 next 265
2025-05-14 06:36:01.398799: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee59500 of size 1024 next 266
2025-05-14 06:36:01.398808: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee59900 of size 1024 next 267
2025-05-14 06:36:01.398817: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee59d00 of size 1024 next 268
2025-05-14 06:36:01.398826: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5a100 of size 4096 next 270
2025-05-14 06:36:01.398836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5b100 of size 4096 next 271
2025-05-14 06:36:01.398845: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5c100 of size 4096 next 272
2025-05-14 06:36:01.398854: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5d100 of size 4096 next 273
2025-05-14 06:36:01.398863: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5e100 of size 4096 next 274
2025-05-14 06:36:01.398872: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5f100 of size 1024 next 275
2025-05-14 06:36:01.398882: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5f500 of size 1024 next 276
2025-05-14 06:36:01.398891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5f900 of size 1024 next 277
2025-05-14 06:36:01.398912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5fd00 of size 1024 next 278
2025-05-14 06:36:01.398922: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee60100 of size 1024 next 279
2025-05-14 06:36:01.398931: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee60500 of size 1024 next 282
2025-05-14 06:36:01.398940: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee60900 of size 1024 next 283
2025-05-14 06:36:01.398950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee60d00 of size 1024 next 284
2025-05-14 06:36:01.398958: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee61100 of size 1024 next 285
2025-05-14 06:36:01.398968: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee61500 of size 1024 next 286
2025-05-14 06:36:01.398976: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee61900 of size 4096 next 288
2025-05-14 06:36:01.398994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee62900 of size 6144 next 27
2025-05-14 06:36:01.399005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee64100 of size 147456 next 26
2025-05-14 06:36:01.399014: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee88100 of size 65536 next 47
2025-05-14 06:36:01.399023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee98100 of size 65536 next 59
2025-05-14 06:36:01.399032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beeea8100 of size 65536 next 65
2025-05-14 06:36:01.399042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beeeb8100 of size 294656 next 18446744073709551615
2025-05-14 06:36:01.399051: I tensorflow/core/common_runtime/bfc_allocator.cc:995]      Summary of in-use Chunks by size: 
2025-05-14 06:36:01.399064: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 59 Chunks of size 256 totalling 14.8KiB
2025-05-14 06:36:01.399076: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB
2025-05-14 06:36:01.399086: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 95 Chunks of size 1024 totalling 95.0KiB
2025-05-14 06:36:01.399097: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB
2025-05-14 06:36:01.399107: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB
2025-05-14 06:36:01.399118: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 38 Chunks of size 4096 totalling 152.0KiB
2025-05-14 06:36:01.399128: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 6144 totalling 6.0KiB
2025-05-14 06:36:01.399138: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB
2025-05-14 06:36:01.399148: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB
2025-05-14 06:36:01.399159: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB
2025-05-14 06:36:01.399170: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB
2025-05-14 06:36:01.399180: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB
2025-05-14 06:36:01.399190: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB
2025-05-14 06:36:01.399200: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB
2025-05-14 06:36:01.399211: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB
2025-05-14 06:36:01.399222: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB
2025-05-14 06:36:01.399232: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB
2025-05-14 06:36:01.399242: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB
2025-05-14 06:36:01.399253: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB
2025-05-14 06:36:01.399263: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 9 Chunks of size 1048576 totalling 9.00MiB
2025-05-14 06:36:01.399273: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 1310720 totalling 5.00MiB
2025-05-14 06:36:01.399283: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB
2025-05-14 06:36:01.399293: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2097152 totalling 2.00MiB
2025-05-14 06:36:01.399303: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 7 Chunks of size 2359296 totalling 15.75MiB
2025-05-14 06:36:01.399313: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2883584 totalling 2.75MiB
2025-05-14 06:36:01.399324: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 3299072 totalling 3.15MiB
2025-05-14 06:36:01.399342: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 45.68MiB
2025-05-14 06:36:01.399352: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 52320000 memory_limit_: 216924160 available bytes: 164604160 curr_region_allocation_bytes_: 67108864
2025-05-14 06:36:01.399366: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: 
Limit:                   216924160
InUse:                    47895296
MaxInUse:                 47895552
NumAllocs:                    1015
MaxAllocSize:              4325376

2025-05-14 06:36:01.399384: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *******__*************************************************************************_***_*************
2025-05-14 06:36:01.399434: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


 Exception in mask_detect :  OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add]
we want to redo the detection
Using TensorFlow backend.
max_time_sub_proc : 3600
erreur pendant la detection
Useless call to update_current_state
in case -12
caffe_path_current : 
About to save ! 1
Inside saveOutput : final : True verbose : False
eke 12-6-18 : saveMask need to be cleaned for new output !
ERROR : mask output needs to be a dictionnary now !
No output to save, continue without doing anything !

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : -12
free memory after detection : 
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 19

############################### TEST detect object  ################################

run mask_detect

Inside batchDatouExec : verbose : False
Catched exception !
Connect or reconnect !
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
List Step Type Loaded in datou : mask_detect
list_input_json : []
origin
BFwe have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.11936712265014648
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Wed May 14 06:36:02 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3000
l 3632 free memory gpu now : 358
wait 20 seconds
l 3637 free memory gpu now : 358  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
2025-05-14 06:36:25.315134: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-05-14 06:36:25.343131: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz
2025-05-14 06:36:25.345204: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f2c90000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-05-14 06:36:25.345266: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-05-14 06:36:25.349347: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-05-14 06:36:25.578722: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x20243d90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-05-14 06:36:25.578768: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-05-14 06:36:25.579234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-05-14 06:36:25.579580: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-05-14 06:36:25.581602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-05-14 06:36:25.583695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-05-14 06:36:25.584045: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-05-14 06:36:25.586359: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-05-14 06:36:25.587412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-05-14 06:36:25.591870: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-05-14 06:36:25.592732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-05-14 06:36:25.592815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-05-14 06:36:25.593276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-05-14 06:36:25.593291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-05-14 06:36:25.593300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-05-14 06:36:25.594010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 206 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
To do loadFromThcl(), then load ParamDescType : thcl454
 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3473
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
2025-05-14 06:36:26.136547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-05-14 06:36:26.136649: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-05-14 06:36:26.136675: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-05-14 06:36:26.136697: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-05-14 06:36:26.136715: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-05-14 06:36:26.136733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-05-14 06:36:26.136751: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-05-14 06:36:26.136769: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-05-14 06:36:26.137584: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-05-14 06:36:26.138533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-05-14 06:36:26.138576: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-05-14 06:36:26.138595: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-05-14 06:36:26.138613: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-05-14 06:36:26.138630: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-05-14 06:36:26.138650: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-05-14 06:36:26.138667: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-05-14 06:36:26.138685: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-05-14 06:36:26.139510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-05-14 06:36:26.139546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-05-14 06:36:26.139557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-05-14 06:36:26.139566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-05-14 06:36:26.140425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 206 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
2025-05-14 06:36:27.326586: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:27.327300: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 28.80M (30199040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:27.327896: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 25.92M (27179264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:27.328394: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 23.33M (24461568 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:27.329023: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 21.00M (22015488 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:27.486650: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:27.487259: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:37.487937: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:37.488433: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-05-14 06:36:37.488453: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25MiB (rounded to 2359296)
Current allocation summary follows.
2025-05-14 06:36:37.488468: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc
2025-05-14 06:36:37.488479: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): 	Total Chunks: 59, Chunks in use: 59. 14.8KiB allocated for chunks. 14.8KiB in use in bin. 8.8KiB client-requested in use in bin.
2025-05-14 06:36:37.488487: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): 	Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin.
2025-05-14 06:36:37.488496: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): 	Total Chunks: 96, Chunks in use: 96. 96.2KiB allocated for chunks. 96.2KiB in use in bin. 96.0KiB client-requested in use in bin.
2025-05-14 06:36:37.488504: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): 	Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin.
2025-05-14 06:36:37.488513: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): 	Total Chunks: 40, Chunks in use: 40. 165.5KiB allocated for chunks. 165.5KiB in use in bin. 160.0KiB client-requested in use in bin.
2025-05-14 06:36:37.488520: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:37.488528: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): 	Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2025-05-14 06:36:37.488537: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): 	Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin.
2025-05-14 06:36:37.488545: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): 	Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin.
2025-05-14 06:36:37.488553: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): 	Total Chunks: 5, Chunks in use: 4. 769.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin.
2025-05-14 06:36:37.488561: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): 	Total Chunks: 8, Chunks in use: 7. 2.16MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin.
2025-05-14 06:36:37.488569: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): 	Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin.
2025-05-14 06:36:37.488577: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): 	Total Chunks: 16, Chunks in use: 14. 17.62MiB allocated for chunks. 15.38MiB in use in bin. 14.00MiB client-requested in use in bin.
2025-05-14 06:36:37.488585: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): 	Total Chunks: 10, Chunks in use: 10. 23.65MiB allocated for chunks. 23.65MiB in use in bin. 22.25MiB client-requested in use in bin.
2025-05-14 06:36:37.488602: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:37.488610: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:37.488617: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:37.488624: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:37.488631: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:37.488638: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:37.488645: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-05-14 06:36:37.488653: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.25MiB was 2.00MiB, Chunk State: 
2025-05-14 06:36:37.488659: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 19814144
2025-05-14 06:36:37.488669: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0000000 of size 2359296 next 245
2025-05-14 06:36:37.488676: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0240000 of size 1048576 next 269
2025-05-14 06:36:37.488682: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab0340000 of size 1310720 next 263
2025-05-14 06:36:37.488688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0480000 of size 2359296 next 262
2025-05-14 06:36:37.488695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab06c0000 of size 1048576 next 287
2025-05-14 06:36:37.488702: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab07c0000 of size 1310720 next 281
2025-05-14 06:36:37.488708: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0900000 of size 2359296 next 280
2025-05-14 06:36:37.488714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0b40000 of size 1048576 next 304
2025-05-14 06:36:37.488720: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0c40000 of size 1310720 next 298
2025-05-14 06:36:37.488727: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0d80000 of size 2359296 next 297
2025-05-14 06:36:37.488733: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab0fc0000 of size 3299072 next 18446744073709551615
2025-05-14 06:36:37.488740: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216
2025-05-14 06:36:37.488746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2000000 of size 2359296 next 181
2025-05-14 06:36:37.488752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2240000 of size 2097152 next 196
2025-05-14 06:36:37.488759: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2440000 of size 1048576 next 216
2025-05-14 06:36:37.488765: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2540000 of size 1310720 next 209
2025-05-14 06:36:37.488771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2680000 of size 2359296 next 208
2025-05-14 06:36:37.488777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab28c0000 of size 1048576 next 234
2025-05-14 06:36:37.488789: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab29c0000 of size 1310720 next 228
2025-05-14 06:36:37.488795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2b00000 of size 2359296 next 227
2025-05-14 06:36:37.488802: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab2d40000 of size 2883584 next 18446744073709551615
2025-05-14 06:36:37.488808: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152
2025-05-14 06:36:37.488814: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4600000 of size 147456 next 55
2025-05-14 06:36:37.488821: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4624000 of size 65536 next 78
2025-05-14 06:36:37.488828: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4634000 of size 4096 next 191
2025-05-14 06:36:37.488834: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4635000 of size 4096 next 192
2025-05-14 06:36:37.488840: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4636000 of size 4096 next 193
2025-05-14 06:36:37.488847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4637000 of size 256 next 194
2025-05-14 06:36:37.488853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4637100 of size 256 next 195
2025-05-14 06:36:37.488859: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4637200 of size 4096 next 197
2025-05-14 06:36:37.488866: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4638200 of size 4096 next 198
2025-05-14 06:36:37.488872: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4639200 of size 4096 next 199
2025-05-14 06:36:37.488878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463a200 of size 4096 next 200
2025-05-14 06:36:37.488884: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463b200 of size 4096 next 201
2025-05-14 06:36:37.488891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463c200 of size 1024 next 202
2025-05-14 06:36:37.488897: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463c600 of size 1024 next 204
2025-05-14 06:36:37.488903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463ca00 of size 1024 next 205
2025-05-14 06:36:37.488910: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463ce00 of size 1024 next 206
2025-05-14 06:36:37.488916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463d200 of size 1024 next 207
2025-05-14 06:36:37.488922: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463d600 of size 1024 next 210
2025-05-14 06:36:37.488928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463da00 of size 1024 next 211
2025-05-14 06:36:37.488935: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463de00 of size 1024 next 212
2025-05-14 06:36:37.488941: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463e200 of size 1024 next 213
2025-05-14 06:36:37.488947: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463e600 of size 1024 next 214
2025-05-14 06:36:37.488954: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463ea00 of size 4096 next 215
2025-05-14 06:36:37.488960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab463fa00 of size 4096 next 217
2025-05-14 06:36:37.488966: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4640a00 of size 4096 next 218
2025-05-14 06:36:37.488972: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4641a00 of size 4096 next 219
2025-05-14 06:36:37.488979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4642a00 of size 4096 next 220
2025-05-14 06:36:37.488985: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4643a00 of size 1024 next 221
2025-05-14 06:36:37.488996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4643e00 of size 1024 next 223
2025-05-14 06:36:37.489002: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4644200 of size 1024 next 224
2025-05-14 06:36:37.489008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4644600 of size 1024 next 225
2025-05-14 06:36:37.489015: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4644a00 of size 1024 next 226
2025-05-14 06:36:37.489021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4644e00 of size 1024 next 229
2025-05-14 06:36:37.489027: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4645200 of size 1024 next 230
2025-05-14 06:36:37.489034: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4645600 of size 1024 next 231
2025-05-14 06:36:37.489040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4645a00 of size 1024 next 232
2025-05-14 06:36:37.489046: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4645e00 of size 1024 next 233
2025-05-14 06:36:37.489053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4646200 of size 7680 next 72
2025-05-14 06:36:37.489059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4648000 of size 147456 next 71
2025-05-14 06:36:37.489065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab466c000 of size 131072 next 87
2025-05-14 06:36:37.489072: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468c000 of size 4096 next 289
2025-05-14 06:36:37.489078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468d000 of size 4096 next 290
2025-05-14 06:36:37.489084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468e000 of size 4096 next 291
2025-05-14 06:36:37.489091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468f000 of size 1024 next 292
2025-05-14 06:36:37.489097: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468f400 of size 1024 next 293
2025-05-14 06:36:37.489103: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468f800 of size 1024 next 294
2025-05-14 06:36:37.489109: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab468fc00 of size 1024 next 295
2025-05-14 06:36:37.489116: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4690000 of size 1024 next 296
2025-05-14 06:36:37.489122: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4690400 of size 1024 next 299
2025-05-14 06:36:37.489128: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4690800 of size 1024 next 300
2025-05-14 06:36:37.489134: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4690c00 of size 1024 next 301
2025-05-14 06:36:37.489141: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4691000 of size 1024 next 302
2025-05-14 06:36:37.489147: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4691400 of size 1024 next 303
2025-05-14 06:36:37.489153: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4691800 of size 4096 next 305
2025-05-14 06:36:37.489160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4692800 of size 4096 next 306
2025-05-14 06:36:37.489166: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4693800 of size 4096 next 307
2025-05-14 06:36:37.489172: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4694800 of size 4096 next 308
2025-05-14 06:36:37.489178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4695800 of size 4096 next 309
2025-05-14 06:36:37.489185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4696800 of size 1024 next 310
2025-05-14 06:36:37.489191: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4696c00 of size 1024 next 311
2025-05-14 06:36:37.489201: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4697000 of size 1024 next 312
2025-05-14 06:36:37.489208: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4697400 of size 1024 next 313
2025-05-14 06:36:37.489214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4697800 of size 1024 next 314
2025-05-14 06:36:37.489221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab4697c00 of size 214016 next 103
2025-05-14 06:36:37.489227: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab46cc000 of size 262144 next 102
2025-05-14 06:36:37.489233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab470c000 of size 999424 next 18446744073709551615
2025-05-14 06:36:37.489240: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304
2025-05-14 06:36:37.489246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4800000 of size 589824 next 96
2025-05-14 06:36:37.489253: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4890000 of size 262144 next 117
2025-05-14 06:36:37.489259: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab48d0000 of size 262144 next 112
2025-05-14 06:36:37.489265: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4910000 of size 524288 next 111
2025-05-14 06:36:37.489272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4990000 of size 262144 next 135
2025-05-14 06:36:37.489278: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab49d0000 of size 327680 next 124
2025-05-14 06:36:37.489285: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4a20000 of size 589824 next 123
2025-05-14 06:36:37.489291: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4ab0000 of size 262144 next 152
2025-05-14 06:36:37.489297: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4af0000 of size 327680 next 142
2025-05-14 06:36:37.489304: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4b40000 of size 786432 next 18446744073709551615
2025-05-14 06:36:37.489311: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608
2025-05-14 06:36:37.489319: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab4c00000 of size 589824 next 160
2025-05-14 06:36:37.489325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4c90000 of size 589824 next 158
2025-05-14 06:36:37.489332: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4d20000 of size 524288 next 171
2025-05-14 06:36:37.489339: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f2ab4da0000 of size 1048576 next 222
2025-05-14 06:36:37.489346: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4ea0000 of size 1048576 next 190
2025-05-14 06:36:37.489353: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab4fa0000 of size 1048576 next 189
2025-05-14 06:36:37.489360: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab50a0000 of size 1048576 next 203
2025-05-14 06:36:37.489366: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab51a0000 of size 1048576 next 251
2025-05-14 06:36:37.489373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2ab52a0000 of size 1441792 next 18446744073709551615
2025-05-14 06:36:37.489380: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576
2025-05-14 06:36:37.489387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00000 of size 1280 next 1
2025-05-14 06:36:37.489394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00500 of size 256 next 5
2025-05-14 06:36:37.489401: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00600 of size 256 next 8
2025-05-14 06:36:37.489408: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00700 of size 256 next 9
2025-05-14 06:36:37.489419: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00800 of size 256 next 10
2025-05-14 06:36:37.489426: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00900 of size 256 next 11
2025-05-14 06:36:37.489433: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00a00 of size 256 next 12
2025-05-14 06:36:37.489440: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00b00 of size 256 next 13
2025-05-14 06:36:37.489446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00c00 of size 256 next 17
2025-05-14 06:36:37.489453: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00d00 of size 256 next 19
2025-05-14 06:36:37.489460: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00e00 of size 256 next 20
2025-05-14 06:36:37.489467: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee00f00 of size 256 next 21
2025-05-14 06:36:37.489473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01000 of size 256 next 22
2025-05-14 06:36:37.489480: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01100 of size 256 next 24
2025-05-14 06:36:37.489487: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01200 of size 256 next 25
2025-05-14 06:36:37.489494: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01300 of size 256 next 23
2025-05-14 06:36:37.489501: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01400 of size 256 next 28
2025-05-14 06:36:37.489507: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01500 of size 256 next 29
2025-05-14 06:36:37.489514: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01600 of size 256 next 30
2025-05-14 06:36:37.489521: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01700 of size 256 next 31
2025-05-14 06:36:37.489528: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01800 of size 256 next 33
2025-05-14 06:36:37.489534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01900 of size 256 next 34
2025-05-14 06:36:37.489541: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01a00 of size 1024 next 32
2025-05-14 06:36:37.489548: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee01e00 of size 1024 next 37
2025-05-14 06:36:37.489555: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee02200 of size 1024 next 38
2025-05-14 06:36:37.489562: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee02600 of size 1024 next 39
2025-05-14 06:36:37.489568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee02a00 of size 1024 next 40
2025-05-14 06:36:37.489575: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee02e00 of size 1024 next 41
2025-05-14 06:36:37.489582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee03200 of size 1024 next 43
2025-05-14 06:36:37.489589: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee03600 of size 1024 next 44
2025-05-14 06:36:37.489595: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee03a00 of size 1024 next 45
2025-05-14 06:36:37.489602: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee03e00 of size 1024 next 46
2025-05-14 06:36:37.489609: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04200 of size 256 next 48
2025-05-14 06:36:37.489616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04300 of size 256 next 49
2025-05-14 06:36:37.489623: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04400 of size 256 next 50
2025-05-14 06:36:37.489629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04500 of size 256 next 51
2025-05-14 06:36:37.489636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04600 of size 256 next 52
2025-05-14 06:36:37.489647: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04700 of size 256 next 53
2025-05-14 06:36:37.489654: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04800 of size 256 next 56
2025-05-14 06:36:37.489661: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04900 of size 256 next 57
2025-05-14 06:36:37.489668: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04a00 of size 256 next 58
2025-05-14 06:36:37.489675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04b00 of size 256 next 14
2025-05-14 06:36:37.489682: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04c00 of size 256 next 15
2025-05-14 06:36:37.489688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04d00 of size 256 next 16
2025-05-14 06:36:37.489695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee04e00 of size 1024 next 60
2025-05-14 06:36:37.489702: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee05200 of size 1024 next 61
2025-05-14 06:36:37.489709: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee05600 of size 1024 next 62
2025-05-14 06:36:37.489716: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee05a00 of size 1024 next 63
2025-05-14 06:36:37.489722: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee05e00 of size 1024 next 64
2025-05-14 06:36:37.489729: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06200 of size 256 next 66
2025-05-14 06:36:37.489736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06300 of size 256 next 67
2025-05-14 06:36:37.489743: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06400 of size 256 next 68
2025-05-14 06:36:37.489750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06500 of size 256 next 69
2025-05-14 06:36:37.489756: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06600 of size 256 next 70
2025-05-14 06:36:37.489763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06700 of size 256 next 73
2025-05-14 06:36:37.489770: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06800 of size 256 next 74
2025-05-14 06:36:37.489777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06900 of size 256 next 75
2025-05-14 06:36:37.489783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06a00 of size 256 next 76
2025-05-14 06:36:37.489790: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06b00 of size 256 next 77
2025-05-14 06:36:37.489797: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee06c00 of size 1024 next 79
2025-05-14 06:36:37.489804: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee07000 of size 1024 next 80
2025-05-14 06:36:37.489811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee07400 of size 1024 next 81
2025-05-14 06:36:37.489817: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee07800 of size 1024 next 82
2025-05-14 06:36:37.489824: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee07c00 of size 1024 next 83
2025-05-14 06:36:37.489831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08000 of size 256 next 85
2025-05-14 06:36:37.489838: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08100 of size 256 next 86
2025-05-14 06:36:37.489845: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08200 of size 512 next 84
2025-05-14 06:36:37.489852: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08400 of size 512 next 88
2025-05-14 06:36:37.489858: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08600 of size 512 next 89
2025-05-14 06:36:37.489865: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08800 of size 512 next 90
2025-05-14 06:36:37.489877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08a00 of size 512 next 91
2025-05-14 06:36:37.489884: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08c00 of size 256 next 93
2025-05-14 06:36:37.489891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08d00 of size 256 next 94
2025-05-14 06:36:37.489898: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee08e00 of size 512 next 92
2025-05-14 06:36:37.489905: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09000 of size 512 next 97
2025-05-14 06:36:37.489912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09200 of size 512 next 98
2025-05-14 06:36:37.489918: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09400 of size 512 next 99
2025-05-14 06:36:37.489925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09600 of size 512 next 2
2025-05-14 06:36:37.489932: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09800 of size 256 next 3
2025-05-14 06:36:37.489939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09900 of size 256 next 4
2025-05-14 06:36:37.489946: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee09a00 of size 16384 next 18
2025-05-14 06:36:37.489952: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0da00 of size 256 next 100
2025-05-14 06:36:37.489959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0db00 of size 256 next 101
2025-05-14 06:36:37.489966: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0dc00 of size 2048 next 104
2025-05-14 06:36:37.489973: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0e400 of size 2048 next 105
2025-05-14 06:36:37.489980: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0ec00 of size 2048 next 106
2025-05-14 06:36:37.489987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0f400 of size 2048 next 107
2025-05-14 06:36:37.489993: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee0fc00 of size 2048 next 108
2025-05-14 06:36:37.490000: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee10400 of size 256 next 109
2025-05-14 06:36:37.490007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee10500 of size 256 next 110
2025-05-14 06:36:37.490014: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee10600 of size 2048 next 113
2025-05-14 06:36:37.490020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee10e00 of size 2048 next 114
2025-05-14 06:36:37.490027: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee11600 of size 2048 next 115
2025-05-14 06:36:37.490040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee11e00 of size 2048 next 116
2025-05-14 06:36:37.490046: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee12600 of size 2048 next 6
2025-05-14 06:36:37.490053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee12e00 of size 37632 next 7
2025-05-14 06:36:37.490060: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c100 of size 512 next 118
2025-05-14 06:36:37.490067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c300 of size 512 next 119
2025-05-14 06:36:37.490074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c500 of size 512 next 120
2025-05-14 06:36:37.490081: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c700 of size 512 next 121
2025-05-14 06:36:37.490087: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1c900 of size 512 next 122
2025-05-14 06:36:37.490094: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1cb00 of size 512 next 125
2025-05-14 06:36:37.490106: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1cd00 of size 512 next 126
2025-05-14 06:36:37.490113: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1cf00 of size 512 next 127
2025-05-14 06:36:37.490120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1d100 of size 512 next 128
2025-05-14 06:36:37.490126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1d300 of size 512 next 129
2025-05-14 06:36:37.490133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1d500 of size 2048 next 130
2025-05-14 06:36:37.490140: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1dd00 of size 2048 next 131
2025-05-14 06:36:37.490147: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1e500 of size 2048 next 132
2025-05-14 06:36:37.490154: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1ed00 of size 2048 next 133
2025-05-14 06:36:37.490160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1f500 of size 2048 next 134
2025-05-14 06:36:37.490167: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1fd00 of size 512 next 136
2025-05-14 06:36:37.490174: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee1ff00 of size 512 next 137
2025-05-14 06:36:37.490181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20100 of size 512 next 138
2025-05-14 06:36:37.490188: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20300 of size 512 next 139
2025-05-14 06:36:37.490194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20500 of size 512 next 140
2025-05-14 06:36:37.490201: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20700 of size 512 next 141
2025-05-14 06:36:37.490208: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20900 of size 512 next 143
2025-05-14 06:36:37.490215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20b00 of size 512 next 144
2025-05-14 06:36:37.490222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20d00 of size 512 next 145
2025-05-14 06:36:37.490228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee20f00 of size 512 next 146
2025-05-14 06:36:37.490235: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee21100 of size 2048 next 147
2025-05-14 06:36:37.490242: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee21900 of size 2048 next 148
2025-05-14 06:36:37.490249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee22100 of size 2048 next 149
2025-05-14 06:36:37.490256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee22900 of size 2048 next 150
2025-05-14 06:36:37.490262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23100 of size 2048 next 151
2025-05-14 06:36:37.490269: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23900 of size 512 next 153
2025-05-14 06:36:37.490276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23b00 of size 512 next 154
2025-05-14 06:36:37.490283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23d00 of size 512 next 155
2025-05-14 06:36:37.490290: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee23f00 of size 512 next 156
2025-05-14 06:36:37.490296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24100 of size 512 next 157
2025-05-14 06:36:37.490303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24300 of size 512 next 161
2025-05-14 06:36:37.490310: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24500 of size 512 next 162
2025-05-14 06:36:37.490317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24700 of size 512 next 163
2025-05-14 06:36:37.490323: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24900 of size 512 next 164
2025-05-14 06:36:37.490335: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24b00 of size 512 next 165
2025-05-14 06:36:37.490342: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee24d00 of size 2048 next 166
2025-05-14 06:36:37.490349: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee25500 of size 2048 next 167
2025-05-14 06:36:37.490355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee25d00 of size 2048 next 168
2025-05-14 06:36:37.490362: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee26500 of size 2048 next 169
2025-05-14 06:36:37.490369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee26d00 of size 2048 next 170
2025-05-14 06:36:37.490376: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee27500 of size 1024 next 172
2025-05-14 06:36:37.490383: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee27900 of size 1024 next 173
2025-05-14 06:36:37.490390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee27d00 of size 1024 next 174
2025-05-14 06:36:37.490396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28100 of size 1024 next 175
2025-05-14 06:36:37.490403: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28500 of size 1024 next 176
2025-05-14 06:36:37.490410: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28900 of size 256 next 178
2025-05-14 06:36:37.490417: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28a00 of size 256 next 179
2025-05-14 06:36:37.490424: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28b00 of size 1024 next 177
2025-05-14 06:36:37.490430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee28f00 of size 1024 next 182
2025-05-14 06:36:37.490437: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee29300 of size 1024 next 183
2025-05-14 06:36:37.490444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee29700 of size 1024 next 184
2025-05-14 06:36:37.490451: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee29b00 of size 1024 next 185
2025-05-14 06:36:37.490458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee29f00 of size 256 next 187
2025-05-14 06:36:37.490465: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee2a000 of size 256 next 188
2025-05-14 06:36:37.490471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee2a100 of size 4096 next 186
2025-05-14 06:36:37.490478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee2b100 of size 4096 next 42
2025-05-14 06:36:37.490485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee2c100 of size 65536 next 36
2025-05-14 06:36:37.490492: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee3c100 of size 65536 next 35
2025-05-14 06:36:37.490499: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee4c100 of size 4096 next 235
2025-05-14 06:36:37.490506: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee4d100 of size 4096 next 236
2025-05-14 06:36:37.490512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee4e100 of size 4096 next 237
2025-05-14 06:36:37.490519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee4f100 of size 4096 next 238
2025-05-14 06:36:37.490526: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee50100 of size 1024 next 239
2025-05-14 06:36:37.490533: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee50500 of size 1024 next 240
2025-05-14 06:36:37.490539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee50900 of size 1024 next 241
2025-05-14 06:36:37.490546: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee50d00 of size 1024 next 242
2025-05-14 06:36:37.490558: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee51100 of size 1024 next 243
2025-05-14 06:36:37.490565: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee51500 of size 1024 next 246
2025-05-14 06:36:37.490572: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee51900 of size 1024 next 247
2025-05-14 06:36:37.490578: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee51d00 of size 1024 next 248
2025-05-14 06:36:37.490585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee52100 of size 1024 next 249
2025-05-14 06:36:37.490592: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee52500 of size 1024 next 250
2025-05-14 06:36:37.490599: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee52900 of size 4096 next 252
2025-05-14 06:36:37.490606: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee53900 of size 4096 next 253
2025-05-14 06:36:37.490612: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee54900 of size 4096 next 254
2025-05-14 06:36:37.490619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee55900 of size 4096 next 255
2025-05-14 06:36:37.490626: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee56900 of size 4096 next 256
2025-05-14 06:36:37.490633: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee57900 of size 1024 next 257
2025-05-14 06:36:37.490640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee57d00 of size 1024 next 258
2025-05-14 06:36:37.490646: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee58100 of size 1024 next 259
2025-05-14 06:36:37.490653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee58500 of size 1024 next 260
2025-05-14 06:36:37.490660: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee58900 of size 1024 next 261
2025-05-14 06:36:37.490667: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee58d00 of size 1024 next 264
2025-05-14 06:36:37.490674: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee59100 of size 1024 next 265
2025-05-14 06:36:37.490680: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee59500 of size 1024 next 266
2025-05-14 06:36:37.490687: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee59900 of size 1024 next 267
2025-05-14 06:36:37.490694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee59d00 of size 1024 next 268
2025-05-14 06:36:37.490701: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5a100 of size 4096 next 270
2025-05-14 06:36:37.490708: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5b100 of size 4096 next 271
2025-05-14 06:36:37.490714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5c100 of size 4096 next 272
2025-05-14 06:36:37.490721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5d100 of size 4096 next 273
2025-05-14 06:36:37.490728: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5e100 of size 4096 next 274
2025-05-14 06:36:37.490735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5f100 of size 1024 next 275
2025-05-14 06:36:37.490742: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5f500 of size 1024 next 276
2025-05-14 06:36:37.490749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5f900 of size 1024 next 277
2025-05-14 06:36:37.490756: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee5fd00 of size 1024 next 278
2025-05-14 06:36:37.490762: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee60100 of size 1024 next 279
2025-05-14 06:36:37.490769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee60500 of size 1024 next 282
2025-05-14 06:36:37.490780: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee60900 of size 1024 next 283
2025-05-14 06:36:37.490787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee60d00 of size 1024 next 284
2025-05-14 06:36:37.490794: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee61100 of size 1024 next 285
2025-05-14 06:36:37.490801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee61500 of size 1024 next 286
2025-05-14 06:36:37.490808: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee61900 of size 4096 next 288
2025-05-14 06:36:37.490815: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee62900 of size 6144 next 27
2025-05-14 06:36:37.490822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee64100 of size 147456 next 26
2025-05-14 06:36:37.490829: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee88100 of size 65536 next 47
2025-05-14 06:36:37.490836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beee98100 of size 65536 next 59
2025-05-14 06:36:37.490842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beeea8100 of size 65536 next 65
2025-05-14 06:36:37.490849: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f2beeeb8100 of size 294656 next 18446744073709551615
2025-05-14 06:36:37.490856: I tensorflow/core/common_runtime/bfc_allocator.cc:995]      Summary of in-use Chunks by size: 
2025-05-14 06:36:37.490865: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 59 Chunks of size 256 totalling 14.8KiB
2025-05-14 06:36:37.490873: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB
2025-05-14 06:36:37.490880: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 95 Chunks of size 1024 totalling 95.0KiB
2025-05-14 06:36:37.490888: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB
2025-05-14 06:36:37.490905: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB
2025-05-14 06:36:37.490914: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 38 Chunks of size 4096 totalling 152.0KiB
2025-05-14 06:36:37.490922: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 6144 totalling 6.0KiB
2025-05-14 06:36:37.490929: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB
2025-05-14 06:36:37.490937: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB
2025-05-14 06:36:37.490944: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB
2025-05-14 06:36:37.490952: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB
2025-05-14 06:36:37.490959: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB
2025-05-14 06:36:37.490967: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB
2025-05-14 06:36:37.490974: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB
2025-05-14 06:36:37.490982: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB
2025-05-14 06:36:37.490989: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB
2025-05-14 06:36:37.490997: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB
2025-05-14 06:36:37.491004: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB
2025-05-14 06:36:37.491012: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB
2025-05-14 06:36:37.491019: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 9 Chunks of size 1048576 totalling 9.00MiB
2025-05-14 06:36:37.491027: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 1310720 totalling 5.00MiB
2025-05-14 06:36:37.491039: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB
2025-05-14 06:36:37.491047: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2097152 totalling 2.00MiB
2025-05-14 06:36:37.491055: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 7 Chunks of size 2359296 totalling 15.75MiB
2025-05-14 06:36:37.491062: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2883584 totalling 2.75MiB
2025-05-14 06:36:37.491070: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 3299072 totalling 3.15MiB
2025-05-14 06:36:37.491077: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 45.68MiB
2025-05-14 06:36:37.491084: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 52320000 memory_limit_: 216924160 available bytes: 164604160 curr_region_allocation_bytes_: 67108864
2025-05-14 06:36:37.491094: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: 
Limit:                   216924160
InUse:                    47895296
MaxInUse:                 47895552
NumAllocs:                    1015
MaxAllocSize:              4325376

2025-05-14 06:36:37.491107: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *******__*************************************************************************_***_*************
2025-05-14 06:36:37.491138: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


 Exception in mask_detect :  OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add]
we want to redo the detection
Using TensorFlow backend.
max_time_sub_proc : 3600
erreur pendant la detection
Useless call to update_current_state
in case -12
caffe_path_current : 
About to save ! 1
Inside saveOutput : final : True verbose : False
eke 12-6-18 : saveMask need to be cleaned for new output !
ERROR : mask output needs to be a dictionnary now !
No output to save, continue without doing anything !

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : -12
ERROR : 'int' object is not subscriptable
reconnect to base !
warning , we can't find thcl infos in json_data
warning , we can't find pdt infos in json_data


#&_# TEST FAILED #&_# : tests/mask_test #&_# 


Error : invalid literal for int() with base 10: "'int' object is not subscriptable"
/home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_6b55ca732dc5789dd54794d418371b3abe63c01d
SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_6b55ca732dc5789dd54794d418371b3abe63c01d','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/May/14052025/python_test3//data_2/data_log/job/2025/May/14052025/python_test3/log-python3----short_python3--v--marlene-06:35:00.txt','mask_detection','unknown');

#&_# END OF TEST #&_# : tests/mask_test #&_# 





#&_# BEGIN OF TEST : tests/datou_test #&_# 


/home/admin/workarea/git/Velours/python/tests/datou_test.py
<module 'tests.datou_test' from '/home/admin/workarea/git/Velours/python/tests/datou_test.py'>
Datou All Test
python version used : 3

############################### TEST sam  ################################
TEST SAM
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4573
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4573
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4573
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : sam
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (1189321094) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 1189321094
download finish for photo 1189321094
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.18940424919128418
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
WARNING : we have an input that is not a photo, we should get rid of it
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:sam
Wed May 14 06:36:37 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1747197397_1576161_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094}
map_photo_id_path_extension : {1189321094: {'path': 'temp/1747197397_1576161_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}}
map_subphoto_mainphoto : {}

Beginning of datou step sam !
pht : 4677
Inside sam : nb paths : 1
ERROR in datou_step_exec, will save and exit !
CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec
    output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2430, in datou_step_exec
    return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 367, in datou_step_sam
    sam.to(device=device)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to
    return self._apply(convert)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 602, in _apply
    param_applied = fn(param)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

[1189321094]
 map_info['map_portfolio_photo'] : {}
 final :  True mtd_id 4573 list_pids : [1189321094]


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 1
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('4573', None, '1189321094', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 100 in test_sam>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 819 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2346 in datou_exec>, 'CUDA error: out of memory\\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.']", '-1', '-1.0', '501120777', '1.0', None)]
time used for this insertion : 0.012041330337524414
 save_final 
ERROR in last step sam, CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

time spend for datou_step_exec : 3.513420343399048
time spend to save output : 0.013284444808959961
total time spend for step 0 : 3.526704788208008
need to delete datou_research and reload, so keep current state 1
need to delete datou_research and reload, so keep current state 1
caffe_path_current : 
About to save ! 2
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : None
ERROR
nb objects espect :  98
nb_objects detect : 0
ERROR sam FAILED

############################### TEST frcnn  ################################
test frcnn
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4184
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4184
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4184
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : frcnn
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (917754606) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 917754606
download finish for photo 917754606
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.11980700492858887
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:frcnn
Wed May 14 06:36:41 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1747197401_1576161_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606}
map_photo_id_path_extension : {917754606: {'path': 'temp/1747197401_1576161_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou step Faster rcnn !
classes : ['background', 'plaque']
pht : 4370
caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'}
To loadFromThcl()
model_param file didn't exist 
model_name : detection_plaque_valcor_010622
model_type : caffe_faster_rcnn
list file need :
['caffemodel', 'test.prototxt']
file exist in s3 :
['caffemodel', 'test.prototxt']
file manque in s3 : 
[]
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0514 06:36:42.954516 1576161 common.cpp:114] Cannot create Cublas handle. Cublas won't be available.
E0514 06:36:42.964488 1576161 common.cpp:121] Cannot create Curand generator. Curand won't be available.
F0514 06:36:42.972164 1576161 syncedmem.hpp:22] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
Command terminated by signal 6
6.46user 6.16system 1:18.74elapsed 16%CPU (0avgtext+0avgdata 1202936maxresident)k
1387752inputs+3416outputs (3347major+589555minor)pagefaults 0swaps