python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120
 import MySQLdb succeeded 
Import error (python version)
python version = 3
warning , we can't find thcl infos in json_data
warning , we can't find pdt infos in json_data
 list_job_run_as_list :  ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel']
python version used : 3
 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')]


#&_# BEGIN OF TEST : tests/mask_test #&_# 


/home/admin/workarea/git/Velours/python/tests/mask_test.py
<module 'tests.mask_test' from '/home/admin/workarea/git/Velours/python/tests/mask_test.py'>
Test mask-detection 
python version used : 3

############################### TEST memory used  ################################
free memory at begining : 
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 322

run mask_detect

Inside batchDatouExec : verbose : False
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
List Step Type Loaded in datou : mask_detect
list_input_json : []
origin
BFwe have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.1402430534362793
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Wed Sep 24 06:35:27 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3000
l 3632 free memory gpu now : 322
wait 20 seconds
l 3637 free memory gpu now : 322  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
/home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
2025-09-24 06:35:51.176718: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-09-24 06:35:51.208566: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz
2025-09-24 06:35:51.211135: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f0d18000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-09-24 06:35:51.211192: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-09-24 06:35:51.215711: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-09-24 06:35:51.435648: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x15f6de10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-09-24 06:35:51.435686: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-09-24 06:35:51.436460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-09-24 06:35:51.438131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-09-24 06:35:51.456968: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-09-24 06:35:51.468929: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-09-24 06:35:51.471872: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-09-24 06:35:51.483901: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-09-24 06:35:51.488225: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-09-24 06:35:51.521052: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-09-24 06:35:51.522130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-09-24 06:35:51.522234: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-09-24 06:35:51.523091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-09-24 06:35:51.523108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-09-24 06:35:51.523119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-09-24 06:35:51.524390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 170 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
To do loadFromThcl(), then load ParamDescType : thcl454
 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3473
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
2025-09-24 06:35:52.352953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-09-24 06:35:52.353037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-09-24 06:35:52.353058: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-09-24 06:35:52.353077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-09-24 06:35:52.353095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-09-24 06:35:52.353112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-09-24 06:35:52.353141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-09-24 06:35:52.353160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-09-24 06:35:52.353958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-09-24 06:35:52.355121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-09-24 06:35:52.355160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-09-24 06:35:52.355178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-09-24 06:35:52.355196: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-09-24 06:35:52.355213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-09-24 06:35:52.355229: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-09-24 06:35:52.355246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-09-24 06:35:52.355263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-09-24 06:35:52.356246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-09-24 06:35:52.356289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-09-24 06:35:52.356301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-09-24 06:35:52.356311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-09-24 06:35:52.357128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 170 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
2025-09-24 06:35:53.979698: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8.00M (8388608 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:35:53.980214: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 7.20M (7549952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:35:53.980780: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.48M (6795008 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:35:53.981347: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.83M (6115584 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:35:53.981909: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.25M (5504256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:35:53.982427: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.72M (4953856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:35:53.982934: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.25M (4458496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:35:54.016497: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:35:54.017018: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:04.018062: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:04.018929: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:04.018961: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25MiB (rounded to 2359296)
Current allocation summary follows.
2025-09-24 06:36:04.018980: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc
2025-09-24 06:36:04.018998: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): 	Total Chunks: 53, Chunks in use: 53. 13.2KiB allocated for chunks. 13.2KiB in use in bin. 8.8KiB client-requested in use in bin.
2025-09-24 06:36:04.019014: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): 	Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin.
2025-09-24 06:36:04.019029: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): 	Total Chunks: 26, Chunks in use: 26. 26.2KiB allocated for chunks. 26.2KiB in use in bin. 26.0KiB client-requested in use in bin.
2025-09-24 06:36:04.019044: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): 	Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin.
2025-09-24 06:36:04.019058: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019072: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): 	Total Chunks: 1, Chunks in use: 0. 14.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019087: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): 	Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2025-09-24 06:36:04.019103: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): 	Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin.
2025-09-24 06:36:04.019118: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): 	Total Chunks: 8, Chunks in use: 6. 560.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin.
2025-09-24 06:36:04.019133: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): 	Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin.
2025-09-24 06:36:04.019148: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): 	Total Chunks: 9, Chunks in use: 7. 2.41MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin.
2025-09-24 06:36:04.019163: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): 	Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin.
2025-09-24 06:36:04.019176: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019190: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): 	Total Chunks: 1, Chunks in use: 0. 2.20MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019203: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019217: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019250: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019264: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019277: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019291: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019304: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:04.019317: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.25MiB was 2.00MiB, Chunk State: 
2025-09-24 06:36:04.019336: I tensorflow/core/common_runtime/bfc_allocator.cc:963]   Size: 2.20MiB | Requested Size: 1B | in_use: 0 | bin_num: 13, prev:   Size: 512.0KiB | Requested Size: 512.0KiB | in_use: 1 | bin_num: -1
2025-09-24 06:36:04.019348: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152
2025-09-24 06:36:04.019364: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34600000 of size 147456 next 55
2025-09-24 06:36:04.019376: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34624000 of size 65536 next 78
2025-09-24 06:36:04.019387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b34634000 of size 81920 next 72
2025-09-24 06:36:04.019399: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34648000 of size 147456 next 71
2025-09-24 06:36:04.019411: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b3466c000 of size 131072 next 87
2025-09-24 06:36:04.019457: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b3468c000 of size 262144 next 103
2025-09-24 06:36:04.019470: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b346cc000 of size 262144 next 102
2025-09-24 06:36:04.019481: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b3470c000 of size 999424 next 18446744073709551615
2025-09-24 06:36:04.019493: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304
2025-09-24 06:36:04.019504: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34800000 of size 589824 next 96
2025-09-24 06:36:04.019516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34890000 of size 262144 next 117
2025-09-24 06:36:04.019527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b348d0000 of size 262144 next 112
2025-09-24 06:36:04.019539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34910000 of size 524288 next 111
2025-09-24 06:36:04.019551: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34990000 of size 262144 next 135
2025-09-24 06:36:04.019563: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b349d0000 of size 327680 next 124
2025-09-24 06:36:04.019574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34a20000 of size 589824 next 123
2025-09-24 06:36:04.019586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34ab0000 of size 262144 next 152
2025-09-24 06:36:04.019597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34af0000 of size 327680 next 142
2025-09-24 06:36:04.019610: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34b40000 of size 786432 next 18446744073709551615
2025-09-24 06:36:04.019632: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4012800
2025-09-24 06:36:04.019644: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b34c00000 of size 589824 next 160
2025-09-24 06:36:04.019656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34c90000 of size 589824 next 158
2025-09-24 06:36:04.019668: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34d20000 of size 524288 next 171
2025-09-24 06:36:04.019680: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b34da0000 of size 2308864 next 18446744073709551615
2025-09-24 06:36:04.019691: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576
2025-09-24 06:36:04.019702: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00000 of size 1280 next 1
2025-09-24 06:36:04.019714: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00500 of size 256 next 5
2025-09-24 06:36:04.019726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00600 of size 256 next 8
2025-09-24 06:36:04.019738: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00700 of size 256 next 9
2025-09-24 06:36:04.019749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00800 of size 256 next 10
2025-09-24 06:36:04.019761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00900 of size 256 next 11
2025-09-24 06:36:04.019772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00a00 of size 256 next 12
2025-09-24 06:36:04.019783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00b00 of size 256 next 13
2025-09-24 06:36:04.019795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00c00 of size 256 next 17
2025-09-24 06:36:04.019806: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00d00 of size 256 next 19
2025-09-24 06:36:04.019818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00e00 of size 256 next 20
2025-09-24 06:36:04.019829: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00f00 of size 256 next 21
2025-09-24 06:36:04.019840: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01000 of size 256 next 22
2025-09-24 06:36:04.019852: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01100 of size 256 next 24
2025-09-24 06:36:04.019863: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01200 of size 256 next 25
2025-09-24 06:36:04.019875: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01300 of size 256 next 23
2025-09-24 06:36:04.019886: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01400 of size 256 next 28
2025-09-24 06:36:04.019898: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01500 of size 256 next 29
2025-09-24 06:36:04.019909: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01600 of size 256 next 30
2025-09-24 06:36:04.019920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01700 of size 256 next 31
2025-09-24 06:36:04.019932: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01800 of size 256 next 33
2025-09-24 06:36:04.019943: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01900 of size 256 next 34
2025-09-24 06:36:04.019955: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01a00 of size 1024 next 32
2025-09-24 06:36:04.019967: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01e00 of size 1024 next 37
2025-09-24 06:36:04.019978: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e02200 of size 1024 next 38
2025-09-24 06:36:04.019990: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e02600 of size 1024 next 39
2025-09-24 06:36:04.020001: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e02a00 of size 1024 next 40
2025-09-24 06:36:04.020021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e02e00 of size 1024 next 41
2025-09-24 06:36:04.020033: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e03200 of size 1024 next 43
2025-09-24 06:36:04.020045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e03600 of size 1024 next 44
2025-09-24 06:36:04.020057: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e03a00 of size 1024 next 45
2025-09-24 06:36:04.020068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e03e00 of size 1024 next 46
2025-09-24 06:36:04.020079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04200 of size 256 next 48
2025-09-24 06:36:04.020091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04300 of size 256 next 49
2025-09-24 06:36:04.020102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04400 of size 256 next 50
2025-09-24 06:36:04.020114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04500 of size 256 next 51
2025-09-24 06:36:04.020125: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04600 of size 256 next 52
2025-09-24 06:36:04.020137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04700 of size 256 next 53
2025-09-24 06:36:04.020148: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04800 of size 256 next 56
2025-09-24 06:36:04.020159: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04900 of size 256 next 57
2025-09-24 06:36:04.020171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04a00 of size 256 next 58
2025-09-24 06:36:04.020183: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04b00 of size 256 next 14
2025-09-24 06:36:04.020194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04c00 of size 256 next 15
2025-09-24 06:36:04.020205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04d00 of size 256 next 16
2025-09-24 06:36:04.020217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04e00 of size 1024 next 60
2025-09-24 06:36:04.020228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e05200 of size 1024 next 61
2025-09-24 06:36:04.020240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e05600 of size 1024 next 62
2025-09-24 06:36:04.020251: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e05a00 of size 1024 next 63
2025-09-24 06:36:04.020263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e05e00 of size 1024 next 64
2025-09-24 06:36:04.020274: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06200 of size 256 next 66
2025-09-24 06:36:04.020285: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06300 of size 256 next 67
2025-09-24 06:36:04.020297: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06400 of size 256 next 68
2025-09-24 06:36:04.020308: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06500 of size 256 next 69
2025-09-24 06:36:04.020320: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06600 of size 256 next 70
2025-09-24 06:36:04.020331: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06700 of size 256 next 73
2025-09-24 06:36:04.020419: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06800 of size 256 next 74
2025-09-24 06:36:04.020442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06900 of size 256 next 75
2025-09-24 06:36:04.020454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06a00 of size 256 next 76
2025-09-24 06:36:04.020466: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06b00 of size 256 next 77
2025-09-24 06:36:04.020477: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06c00 of size 1024 next 79
2025-09-24 06:36:04.020506: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e07000 of size 1024 next 80
2025-09-24 06:36:04.020518: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e07400 of size 1024 next 81
2025-09-24 06:36:04.020529: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e07800 of size 1024 next 82
2025-09-24 06:36:04.020540: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e07c00 of size 1024 next 83
2025-09-24 06:36:04.020552: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08000 of size 256 next 85
2025-09-24 06:36:04.020563: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08100 of size 256 next 86
2025-09-24 06:36:04.020575: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08200 of size 512 next 84
2025-09-24 06:36:04.020586: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08400 of size 512 next 88
2025-09-24 06:36:04.020601: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08600 of size 512 next 89
2025-09-24 06:36:04.020616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08800 of size 512 next 90
2025-09-24 06:36:04.020630: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08a00 of size 512 next 91
2025-09-24 06:36:04.020645: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08c00 of size 256 next 93
2025-09-24 06:36:04.020660: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08d00 of size 256 next 94
2025-09-24 06:36:04.020675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08e00 of size 512 next 92
2025-09-24 06:36:04.020690: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09000 of size 512 next 97
2025-09-24 06:36:04.020704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09200 of size 512 next 98
2025-09-24 06:36:04.020719: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09400 of size 512 next 99
2025-09-24 06:36:04.020733: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09600 of size 512 next 2
2025-09-24 06:36:04.020748: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09800 of size 256 next 3
2025-09-24 06:36:04.020763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09900 of size 256 next 4
2025-09-24 06:36:04.020778: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09a00 of size 16384 next 18
2025-09-24 06:36:04.020793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0da00 of size 256 next 100
2025-09-24 06:36:04.020807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0db00 of size 256 next 101
2025-09-24 06:36:04.020822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0dc00 of size 2048 next 104
2025-09-24 06:36:04.020837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0e400 of size 2048 next 105
2025-09-24 06:36:04.020850: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0ec00 of size 2048 next 106
2025-09-24 06:36:04.020862: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0f400 of size 2048 next 107
2025-09-24 06:36:04.020873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0fc00 of size 2048 next 108
2025-09-24 06:36:04.020884: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e10400 of size 256 next 109
2025-09-24 06:36:04.020896: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e10500 of size 256 next 110
2025-09-24 06:36:04.020907: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e10600 of size 2048 next 113
2025-09-24 06:36:04.020918: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e10e00 of size 2048 next 114
2025-09-24 06:36:04.024176: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e11600 of size 2048 next 115
2025-09-24 06:36:04.024195: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e11e00 of size 2048 next 116
2025-09-24 06:36:04.024206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e12600 of size 2048 next 6
2025-09-24 06:36:04.024218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e12e00 of size 37632 next 7
2025-09-24 06:36:04.024229: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c100 of size 512 next 118
2025-09-24 06:36:04.024239: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c300 of size 512 next 119
2025-09-24 06:36:04.024250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c500 of size 512 next 120
2025-09-24 06:36:04.024260: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c700 of size 512 next 121
2025-09-24 06:36:04.024270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c900 of size 512 next 122
2025-09-24 06:36:04.024281: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1cb00 of size 512 next 125
2025-09-24 06:36:04.024292: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1cd00 of size 512 next 126
2025-09-24 06:36:04.024303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1cf00 of size 512 next 127
2025-09-24 06:36:04.024313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1d100 of size 512 next 128
2025-09-24 06:36:04.024324: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1d300 of size 512 next 129
2025-09-24 06:36:04.024334: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1d500 of size 2048 next 130
2025-09-24 06:36:04.024361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1dd00 of size 2048 next 131
2025-09-24 06:36:04.024372: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1e500 of size 2048 next 132
2025-09-24 06:36:04.024383: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1ed00 of size 2048 next 133
2025-09-24 06:36:04.024394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1f500 of size 2048 next 134
2025-09-24 06:36:04.024404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1fd00 of size 512 next 136
2025-09-24 06:36:04.024414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1ff00 of size 512 next 137
2025-09-24 06:36:04.024425: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20100 of size 512 next 138
2025-09-24 06:36:04.024436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20300 of size 512 next 139
2025-09-24 06:36:04.024446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20500 of size 512 next 140
2025-09-24 06:36:04.024457: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20700 of size 512 next 141
2025-09-24 06:36:04.024467: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20900 of size 512 next 143
2025-09-24 06:36:04.024477: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20b00 of size 512 next 144
2025-09-24 06:36:04.024488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20d00 of size 512 next 145
2025-09-24 06:36:04.024498: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20f00 of size 512 next 146
2025-09-24 06:36:04.024509: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e21100 of size 2048 next 147
2025-09-24 06:36:04.024520: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e21900 of size 2048 next 148
2025-09-24 06:36:04.024530: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e22100 of size 2048 next 149
2025-09-24 06:36:04.024541: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e22900 of size 2048 next 150
2025-09-24 06:36:04.024560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23100 of size 2048 next 151
2025-09-24 06:36:04.024572: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23900 of size 512 next 153
2025-09-24 06:36:04.024582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23b00 of size 512 next 154
2025-09-24 06:36:04.024593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23d00 of size 512 next 155
2025-09-24 06:36:04.024603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23f00 of size 512 next 156
2025-09-24 06:36:04.024614: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24100 of size 512 next 157
2025-09-24 06:36:04.024624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24300 of size 512 next 161
2025-09-24 06:36:04.024635: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24500 of size 512 next 162
2025-09-24 06:36:04.024646: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24700 of size 512 next 163
2025-09-24 06:36:04.024656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24900 of size 512 next 164
2025-09-24 06:36:04.024666: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24b00 of size 512 next 165
2025-09-24 06:36:04.024676: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24d00 of size 2048 next 166
2025-09-24 06:36:04.024687: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e25500 of size 2048 next 167
2025-09-24 06:36:04.024697: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e25d00 of size 2048 next 168
2025-09-24 06:36:04.024708: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e26500 of size 2048 next 169
2025-09-24 06:36:04.024718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e26d00 of size 2048 next 170
2025-09-24 06:36:04.024729: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e27500 of size 1024 next 172
2025-09-24 06:36:04.024740: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e27900 of size 1024 next 173
2025-09-24 06:36:04.024750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e27d00 of size 1024 next 174
2025-09-24 06:36:04.024761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e28100 of size 1024 next 175
2025-09-24 06:36:04.024772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e28500 of size 1024 next 176
2025-09-24 06:36:04.024783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0c70e28900 of size 14336 next 42
2025-09-24 06:36:04.024793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e2c100 of size 65536 next 36
2025-09-24 06:36:04.024804: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e3c100 of size 65536 next 35
2025-09-24 06:36:04.024814: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0c70e4c100 of size 98304 next 27
2025-09-24 06:36:04.024825: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e64100 of size 147456 next 26
2025-09-24 06:36:04.024837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e88100 of size 65536 next 47
2025-09-24 06:36:04.024848: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e98100 of size 65536 next 59
2025-09-24 06:36:04.024859: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70ea8100 of size 65536 next 65
2025-09-24 06:36:04.024870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70eb8100 of size 294656 next 18446744073709551615
2025-09-24 06:36:04.024881: I tensorflow/core/common_runtime/bfc_allocator.cc:995]      Summary of in-use Chunks by size: 
2025-09-24 06:36:04.024897: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 53 Chunks of size 256 totalling 13.2KiB
2025-09-24 06:36:04.024919: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB
2025-09-24 06:36:04.024931: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1024 totalling 25.0KiB
2025-09-24 06:36:04.024943: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB
2025-09-24 06:36:04.024955: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB
2025-09-24 06:36:04.024966: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB
2025-09-24 06:36:04.024978: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB
2025-09-24 06:36:04.024990: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB
2025-09-24 06:36:04.025002: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB
2025-09-24 06:36:04.025017: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB
2025-09-24 06:36:04.025028: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB
2025-09-24 06:36:04.025041: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB
2025-09-24 06:36:04.025053: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB
2025-09-24 06:36:04.025064: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB
2025-09-24 06:36:04.025076: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB
2025-09-24 06:36:04.025088: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB
2025-09-24 06:36:04.025100: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 6.42MiB
2025-09-24 06:36:04.025111: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 11352832 memory_limit_: 178847744 available bytes: 167494912 curr_region_allocation_bytes_: 16777216
2025-09-24 06:36:04.025126: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: 
Limit:                   178847744
InUse:                     6735872
MaxInUse:                  8320000
NumAllocs:                     544
MaxAllocSize:               999424

2025-09-24 06:36:04.025143: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ******_***________********__**************************xx____***********___________________**********
2025-09-24 06:36:04.025527: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at random_op.cc:77 : Resource exhausted: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


 Exception in mask_detect :  OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform]
we want to redo the detection
Using TensorFlow backend.
max_time_sub_proc : 3600
erreur pendant la detection
Useless call to update_current_state
in case -12
caffe_path_current : 
About to save ! 1
Inside saveOutput : final : True verbose : False
eke 12-6-18 : saveMask need to be cleaned for new output !
ERROR : mask output needs to be a dictionnary now !
No output to save, continue without doing anything !

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : -12
free memory after detection : 
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 23

############################### TEST detect object  ################################

run mask_detect

Inside batchDatouExec : verbose : False
Catched exception !
Connect or reconnect !
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
List Step Type Loaded in datou : mask_detect
list_input_json : []
origin
BFwe have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.1911616325378418
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Wed Sep 24 06:36:05 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3000
l 3632 free memory gpu now : 322
wait 20 seconds
l 3637 free memory gpu now : 322  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
2025-09-24 06:36:28.854573: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-09-24 06:36:28.880643: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz
2025-09-24 06:36:28.882493: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f0d18000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-09-24 06:36:28.882521: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-09-24 06:36:28.886299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-09-24 06:36:29.083734: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x15df0e50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-09-24 06:36:29.083790: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-09-24 06:36:29.084661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-09-24 06:36:29.085441: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-09-24 06:36:29.091152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-09-24 06:36:29.095349: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-09-24 06:36:29.096647: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-09-24 06:36:29.100536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-09-24 06:36:29.101831: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-09-24 06:36:29.110908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-09-24 06:36:29.111691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-09-24 06:36:29.111748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-09-24 06:36:29.112172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-09-24 06:36:29.112186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-09-24 06:36:29.112194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-09-24 06:36:29.113459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 170 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
To do loadFromThcl(), then load ParamDescType : thcl454
 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3473
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
2025-09-24 06:36:29.782298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-09-24 06:36:29.782370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-09-24 06:36:29.782391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-09-24 06:36:29.782409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-09-24 06:36:29.782427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-09-24 06:36:29.782444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-09-24 06:36:29.782462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-09-24 06:36:29.782480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-09-24 06:36:29.783262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-09-24 06:36:29.784195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-09-24 06:36:29.784230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-09-24 06:36:29.784249: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-09-24 06:36:29.784267: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-09-24 06:36:29.784284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-09-24 06:36:29.784301: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-09-24 06:36:29.784318: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-09-24 06:36:29.784381: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-09-24 06:36:29.785167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-09-24 06:36:29.785198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-09-24 06:36:29.785208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-09-24 06:36:29.785217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-09-24 06:36:29.786063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 170 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
2025-09-24 06:36:31.478633: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8.00M (8388608 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:31.479234: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 7.20M (7549952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:31.479775: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.48M (6795008 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:31.480370: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.83M (6115584 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:31.480955: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.25M (5504256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:31.481531: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.72M (4953856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:31.482383: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.25M (4458496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:31.517317: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:31.517836: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:41.518453: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:41.518947: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-09-24 06:36:41.518967: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25MiB (rounded to 2359296)
Current allocation summary follows.
2025-09-24 06:36:41.522311: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc
2025-09-24 06:36:41.522365: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): 	Total Chunks: 53, Chunks in use: 53. 13.2KiB allocated for chunks. 13.2KiB in use in bin. 8.8KiB client-requested in use in bin.
2025-09-24 06:36:41.522382: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): 	Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin.
2025-09-24 06:36:41.522398: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): 	Total Chunks: 26, Chunks in use: 26. 26.2KiB allocated for chunks. 26.2KiB in use in bin. 26.0KiB client-requested in use in bin.
2025-09-24 06:36:41.522413: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): 	Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin.
2025-09-24 06:36:41.522446: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522461: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): 	Total Chunks: 1, Chunks in use: 0. 14.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522476: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): 	Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2025-09-24 06:36:41.522491: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): 	Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin.
2025-09-24 06:36:41.522506: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): 	Total Chunks: 8, Chunks in use: 6. 560.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin.
2025-09-24 06:36:41.522521: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): 	Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin.
2025-09-24 06:36:41.522536: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): 	Total Chunks: 9, Chunks in use: 7. 2.41MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin.
2025-09-24 06:36:41.522551: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): 	Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin.
2025-09-24 06:36:41.522564: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522578: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): 	Total Chunks: 1, Chunks in use: 0. 2.20MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522591: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522604: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522617: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522630: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522643: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522656: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522669: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-09-24 06:36:41.522683: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.25MiB was 2.00MiB, Chunk State: 
2025-09-24 06:36:41.522702: I tensorflow/core/common_runtime/bfc_allocator.cc:963]   Size: 2.20MiB | Requested Size: 1B | in_use: 0 | bin_num: 13, prev:   Size: 512.0KiB | Requested Size: 512.0KiB | in_use: 1 | bin_num: -1
2025-09-24 06:36:41.522724: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152
2025-09-24 06:36:41.522740: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34600000 of size 147456 next 55
2025-09-24 06:36:41.522752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34624000 of size 65536 next 78
2025-09-24 06:36:41.522763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b34634000 of size 81920 next 72
2025-09-24 06:36:41.522775: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34648000 of size 147456 next 71
2025-09-24 06:36:41.522787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b3466c000 of size 131072 next 87
2025-09-24 06:36:41.522799: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b3468c000 of size 262144 next 103
2025-09-24 06:36:41.522810: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b346cc000 of size 262144 next 102
2025-09-24 06:36:41.522822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b3470c000 of size 999424 next 18446744073709551615
2025-09-24 06:36:41.522833: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304
2025-09-24 06:36:41.522845: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34800000 of size 589824 next 96
2025-09-24 06:36:41.522857: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34890000 of size 262144 next 117
2025-09-24 06:36:41.522868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b348d0000 of size 262144 next 112
2025-09-24 06:36:41.522880: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34910000 of size 524288 next 111
2025-09-24 06:36:41.522892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34990000 of size 262144 next 135
2025-09-24 06:36:41.522904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b349d0000 of size 327680 next 124
2025-09-24 06:36:41.522915: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34a20000 of size 589824 next 123
2025-09-24 06:36:41.522927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34ab0000 of size 262144 next 152
2025-09-24 06:36:41.522939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34af0000 of size 327680 next 142
2025-09-24 06:36:41.522951: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34b40000 of size 786432 next 18446744073709551615
2025-09-24 06:36:41.522962: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4012800
2025-09-24 06:36:41.522974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b34c00000 of size 589824 next 160
2025-09-24 06:36:41.522985: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34c90000 of size 589824 next 158
2025-09-24 06:36:41.522998: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0b34d20000 of size 524288 next 171
2025-09-24 06:36:41.523012: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0b34da0000 of size 2308864 next 18446744073709551615
2025-09-24 06:36:41.523024: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576
2025-09-24 06:36:41.523036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00000 of size 1280 next 1
2025-09-24 06:36:41.523049: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00500 of size 256 next 5
2025-09-24 06:36:41.523062: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00600 of size 256 next 8
2025-09-24 06:36:41.523073: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00700 of size 256 next 9
2025-09-24 06:36:41.523085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00800 of size 256 next 10
2025-09-24 06:36:41.523096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00900 of size 256 next 11
2025-09-24 06:36:41.523116: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00a00 of size 256 next 12
2025-09-24 06:36:41.523128: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00b00 of size 256 next 13
2025-09-24 06:36:41.523140: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00c00 of size 256 next 17
2025-09-24 06:36:41.523151: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00d00 of size 256 next 19
2025-09-24 06:36:41.523163: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00e00 of size 256 next 20
2025-09-24 06:36:41.523174: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e00f00 of size 256 next 21
2025-09-24 06:36:41.523186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01000 of size 256 next 22
2025-09-24 06:36:41.523197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01100 of size 256 next 24
2025-09-24 06:36:41.523209: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01200 of size 256 next 25
2025-09-24 06:36:41.523220: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01300 of size 256 next 23
2025-09-24 06:36:41.523232: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01400 of size 256 next 28
2025-09-24 06:36:41.523243: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01500 of size 256 next 29
2025-09-24 06:36:41.523255: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01600 of size 256 next 30
2025-09-24 06:36:41.523266: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01700 of size 256 next 31
2025-09-24 06:36:41.523278: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01800 of size 256 next 33
2025-09-24 06:36:41.523289: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01900 of size 256 next 34
2025-09-24 06:36:41.523301: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01a00 of size 1024 next 32
2025-09-24 06:36:41.523313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e01e00 of size 1024 next 37
2025-09-24 06:36:41.523324: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e02200 of size 1024 next 38
2025-09-24 06:36:41.523336: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e02600 of size 1024 next 39
2025-09-24 06:36:41.523347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e02a00 of size 1024 next 40
2025-09-24 06:36:41.523359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e02e00 of size 1024 next 41
2025-09-24 06:36:41.523370: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e03200 of size 1024 next 43
2025-09-24 06:36:41.523382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e03600 of size 1024 next 44
2025-09-24 06:36:41.523393: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e03a00 of size 1024 next 45
2025-09-24 06:36:41.523405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e03e00 of size 1024 next 46
2025-09-24 06:36:41.523416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04200 of size 256 next 48
2025-09-24 06:36:41.523428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04300 of size 256 next 49
2025-09-24 06:36:41.523439: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04400 of size 256 next 50
2025-09-24 06:36:41.523450: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04500 of size 256 next 51
2025-09-24 06:36:41.523462: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04600 of size 256 next 52
2025-09-24 06:36:41.523473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04700 of size 256 next 53
2025-09-24 06:36:41.523485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04800 of size 256 next 56
2025-09-24 06:36:41.523504: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04900 of size 256 next 57
2025-09-24 06:36:41.523516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04a00 of size 256 next 58
2025-09-24 06:36:41.523527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04b00 of size 256 next 14
2025-09-24 06:36:41.523539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04c00 of size 256 next 15
2025-09-24 06:36:41.523550: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04d00 of size 256 next 16
2025-09-24 06:36:41.523562: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e04e00 of size 1024 next 60
2025-09-24 06:36:41.523573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e05200 of size 1024 next 61
2025-09-24 06:36:41.523585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e05600 of size 1024 next 62
2025-09-24 06:36:41.523596: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e05a00 of size 1024 next 63
2025-09-24 06:36:41.523608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e05e00 of size 1024 next 64
2025-09-24 06:36:41.523619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06200 of size 256 next 66
2025-09-24 06:36:41.523631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06300 of size 256 next 67
2025-09-24 06:36:41.523642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06400 of size 256 next 68
2025-09-24 06:36:41.523654: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06500 of size 256 next 69
2025-09-24 06:36:41.523665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06600 of size 256 next 70
2025-09-24 06:36:41.523677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06700 of size 256 next 73
2025-09-24 06:36:41.523688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06800 of size 256 next 74
2025-09-24 06:36:41.523700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06900 of size 256 next 75
2025-09-24 06:36:41.523711: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06a00 of size 256 next 76
2025-09-24 06:36:41.523723: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06b00 of size 256 next 77
2025-09-24 06:36:41.523734: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e06c00 of size 1024 next 79
2025-09-24 06:36:41.523746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e07000 of size 1024 next 80
2025-09-24 06:36:41.523757: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e07400 of size 1024 next 81
2025-09-24 06:36:41.523769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e07800 of size 1024 next 82
2025-09-24 06:36:41.523780: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e07c00 of size 1024 next 83
2025-09-24 06:36:41.523792: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08000 of size 256 next 85
2025-09-24 06:36:41.523803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08100 of size 256 next 86
2025-09-24 06:36:41.523815: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08200 of size 512 next 84
2025-09-24 06:36:41.523827: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08400 of size 512 next 88
2025-09-24 06:36:41.523838: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08600 of size 512 next 89
2025-09-24 06:36:41.523850: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08800 of size 512 next 90
2025-09-24 06:36:41.523861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08a00 of size 512 next 91
2025-09-24 06:36:41.523873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08c00 of size 256 next 93
2025-09-24 06:36:41.523892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08d00 of size 256 next 94
2025-09-24 06:36:41.523904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e08e00 of size 512 next 92
2025-09-24 06:36:41.523915: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09000 of size 512 next 97
2025-09-24 06:36:41.523927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09200 of size 512 next 98
2025-09-24 06:36:41.523938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09400 of size 512 next 99
2025-09-24 06:36:41.523950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09600 of size 512 next 2
2025-09-24 06:36:41.523961: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09800 of size 256 next 3
2025-09-24 06:36:41.523973: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09900 of size 256 next 4
2025-09-24 06:36:41.523984: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e09a00 of size 16384 next 18
2025-09-24 06:36:41.523996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0da00 of size 256 next 100
2025-09-24 06:36:41.524007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0db00 of size 256 next 101
2025-09-24 06:36:41.524019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0dc00 of size 2048 next 104
2025-09-24 06:36:41.524030: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0e400 of size 2048 next 105
2025-09-24 06:36:41.524042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0ec00 of size 2048 next 106
2025-09-24 06:36:41.524053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0f400 of size 2048 next 107
2025-09-24 06:36:41.524065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e0fc00 of size 2048 next 108
2025-09-24 06:36:41.524076: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e10400 of size 256 next 109
2025-09-24 06:36:41.524088: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e10500 of size 256 next 110
2025-09-24 06:36:41.524099: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e10600 of size 2048 next 113
2025-09-24 06:36:41.524111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e10e00 of size 2048 next 114
2025-09-24 06:36:41.524122: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e11600 of size 2048 next 115
2025-09-24 06:36:41.524133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e11e00 of size 2048 next 116
2025-09-24 06:36:41.524145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e12600 of size 2048 next 6
2025-09-24 06:36:41.524156: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e12e00 of size 37632 next 7
2025-09-24 06:36:41.524168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c100 of size 512 next 118
2025-09-24 06:36:41.524179: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c300 of size 512 next 119
2025-09-24 06:36:41.524191: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c500 of size 512 next 120
2025-09-24 06:36:41.524202: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c700 of size 512 next 121
2025-09-24 06:36:41.524214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1c900 of size 512 next 122
2025-09-24 06:36:41.524225: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1cb00 of size 512 next 125
2025-09-24 06:36:41.524237: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1cd00 of size 512 next 126
2025-09-24 06:36:41.524248: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1cf00 of size 512 next 127
2025-09-24 06:36:41.524273: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1d100 of size 512 next 128
2025-09-24 06:36:41.524285: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1d300 of size 512 next 129
2025-09-24 06:36:41.524297: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1d500 of size 2048 next 130
2025-09-24 06:36:41.524308: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1dd00 of size 2048 next 131
2025-09-24 06:36:41.524320: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1e500 of size 2048 next 132
2025-09-24 06:36:41.524331: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1ed00 of size 2048 next 133
2025-09-24 06:36:41.524415: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1f500 of size 2048 next 134
2025-09-24 06:36:41.524432: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1fd00 of size 512 next 136
2025-09-24 06:36:41.524444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e1ff00 of size 512 next 137
2025-09-24 06:36:41.524455: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20100 of size 512 next 138
2025-09-24 06:36:41.524467: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20300 of size 512 next 139
2025-09-24 06:36:41.524478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20500 of size 512 next 140
2025-09-24 06:36:41.524489: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20700 of size 512 next 141
2025-09-24 06:36:41.524502: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20900 of size 512 next 143
2025-09-24 06:36:41.524513: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20b00 of size 512 next 144
2025-09-24 06:36:41.524524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20d00 of size 512 next 145
2025-09-24 06:36:41.524536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e20f00 of size 512 next 146
2025-09-24 06:36:41.524547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e21100 of size 2048 next 147
2025-09-24 06:36:41.524559: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e21900 of size 2048 next 148
2025-09-24 06:36:41.524570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e22100 of size 2048 next 149
2025-09-24 06:36:41.524581: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e22900 of size 2048 next 150
2025-09-24 06:36:41.524593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23100 of size 2048 next 151
2025-09-24 06:36:41.524604: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23900 of size 512 next 153
2025-09-24 06:36:41.524616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23b00 of size 512 next 154
2025-09-24 06:36:41.524627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23d00 of size 512 next 155
2025-09-24 06:36:41.524638: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e23f00 of size 512 next 156
2025-09-24 06:36:41.524650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24100 of size 512 next 157
2025-09-24 06:36:41.524661: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24300 of size 512 next 161
2025-09-24 06:36:41.524673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24500 of size 512 next 162
2025-09-24 06:36:41.524684: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24700 of size 512 next 163
2025-09-24 06:36:41.524695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24900 of size 512 next 164
2025-09-24 06:36:41.524707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24b00 of size 512 next 165
2025-09-24 06:36:41.524719: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e24d00 of size 2048 next 166
2025-09-24 06:36:41.524739: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e25500 of size 2048 next 167
2025-09-24 06:36:41.524751: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e25d00 of size 2048 next 168
2025-09-24 06:36:41.524762: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e26500 of size 2048 next 169
2025-09-24 06:36:41.524774: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e26d00 of size 2048 next 170
2025-09-24 06:36:41.524785: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e27500 of size 1024 next 172
2025-09-24 06:36:41.524797: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e27900 of size 1024 next 173
2025-09-24 06:36:41.524808: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e27d00 of size 1024 next 174
2025-09-24 06:36:41.524820: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e28100 of size 1024 next 175
2025-09-24 06:36:41.524831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e28500 of size 1024 next 176
2025-09-24 06:36:41.524843: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0c70e28900 of size 14336 next 42
2025-09-24 06:36:41.524854: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e2c100 of size 65536 next 36
2025-09-24 06:36:41.524866: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e3c100 of size 65536 next 35
2025-09-24 06:36:41.524877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free  at 7f0c70e4c100 of size 98304 next 27
2025-09-24 06:36:41.524889: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e64100 of size 147456 next 26
2025-09-24 06:36:41.524900: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e88100 of size 65536 next 47
2025-09-24 06:36:41.524912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70e98100 of size 65536 next 59
2025-09-24 06:36:41.524923: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70ea8100 of size 65536 next 65
2025-09-24 06:36:41.524935: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f0c70eb8100 of size 294656 next 18446744073709551615
2025-09-24 06:36:41.524947: I tensorflow/core/common_runtime/bfc_allocator.cc:995]      Summary of in-use Chunks by size: 
2025-09-24 06:36:41.524962: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 53 Chunks of size 256 totalling 13.2KiB
2025-09-24 06:36:41.524976: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB
2025-09-24 06:36:41.524989: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1024 totalling 25.0KiB
2025-09-24 06:36:41.525002: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB
2025-09-24 06:36:41.525014: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB
2025-09-24 06:36:41.525028: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB
2025-09-24 06:36:41.525041: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB
2025-09-24 06:36:41.525054: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB
2025-09-24 06:36:41.525067: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB
2025-09-24 06:36:41.525080: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB
2025-09-24 06:36:41.525092: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB
2025-09-24 06:36:41.525106: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB
2025-09-24 06:36:41.525119: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB
2025-09-24 06:36:41.525140: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB
2025-09-24 06:36:41.525153: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB
2025-09-24 06:36:41.525166: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB
2025-09-24 06:36:41.525179: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 6.42MiB
2025-09-24 06:36:41.525191: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 11352832 memory_limit_: 178847744 available bytes: 167494912 curr_region_allocation_bytes_: 16777216
2025-09-24 06:36:41.525206: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: 
Limit:                   178847744
InUse:                     6735872
MaxInUse:                  8320000
NumAllocs:                     544
MaxAllocSize:               999424

2025-09-24 06:36:41.525225: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ******_***________********__**************************xx____***********___________________**********
2025-09-24 06:36:41.525636: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at random_op.cc:77 : Resource exhausted: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


 Exception in mask_detect :  OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform]
we want to redo the detection
Using TensorFlow backend.
max_time_sub_proc : 3600
erreur pendant la detection
Useless call to update_current_state
in case -12
caffe_path_current : 
About to save ! 1
Inside saveOutput : final : True verbose : False
eke 12-6-18 : saveMask need to be cleaned for new output !
ERROR : mask output needs to be a dictionnary now !
No output to save, continue without doing anything !

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : -12
ERROR : 'int' object is not subscriptable
reconnect to base !
warning , we can't find thcl infos in json_data
warning , we can't find pdt infos in json_data


#&_# TEST FAILED #&_# : tests/mask_test #&_# 


Error : invalid literal for int() with base 10: "'int' object is not subscriptable"
/home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_226dbe25fdb6bbef932c61822f28a0da16637277
SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_226dbe25fdb6bbef932c61822f28a0da16637277','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/September/24092025/python_test3//data_4/data_log/job/2025/September/24092025/python_test3/log-python3----short_python3--v--marlene-06:35:01.txt','mask_detection','unknown');

#&_# END OF TEST #&_# : tests/mask_test #&_# 





#&_# BEGIN OF TEST : tests/datou_test #&_# 


/home/admin/workarea/git/Velours/python/tests/datou_test.py
<module 'tests.datou_test' from '/home/admin/workarea/git/Velours/python/tests/datou_test.py'>
Datou All Test
python version used : 3

############################### TEST sam  ################################
TEST SAM
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4573
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4573
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4573
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : sam
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (1189321094) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 1189321094
download finish for photo 1189321094
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.2588067054748535
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
WARNING : we have an input that is not a photo, we should get rid of it
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:sam
Wed Sep 24 06:36:42 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1758688601_828627_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094}
map_photo_id_path_extension : {1189321094: {'path': 'temp/1758688601_828627_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}}
map_subphoto_mainphoto : {}

Beginning of datou step sam !
pht : 4677
Inside sam : nb paths : 1
ERROR in datou_step_exec, will save and exit !
CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2332, in datou_exec
    output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2433, in datou_step_exec
    return lib_process.datou_step_sam(param, json_param, args, cache, context, map_info, verbose)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 367, in datou_step_sam
    sam.to(device=device)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to
    return self._apply(convert)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 602, in _apply
    param_applied = fn(param)
  File "/home/admin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

[1189321094]
 map_info['map_portfolio_photo'] : {}
 final :  True mtd_id 4573 list_pids : [1189321094]


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 1
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('4573', None, '1189321094', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 100 in test_sam>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 820 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2349 in datou_exec>, 'CUDA error: out of memory\\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.']", '-1', '-1.0', '501120777', '1.0', None)]
time used for this insertion : 0.01949143409729004
 save_final 
ERROR in last step sam, CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

time spend for datou_step_exec : 12.760464906692505
time spend to save output : 0.020833969116210938
total time spend for step 0 : 12.781298875808716
need to delete datou_research and reload, so keep current state 1
need to delete datou_research and reload, so keep current state 1
caffe_path_current : 
About to save ! 2
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : None
ERROR
nb objects espect :  98
nb_objects detect : 0
ERROR sam FAILED

############################### TEST frcnn  ################################
test frcnn
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4184
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4184
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4184
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : frcnn
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (917754606) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 917754606
download finish for photo 917754606
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.16180419921875
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:frcnn
Wed Sep 24 06:36:55 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1758688614_828627_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606}
map_photo_id_path_extension : {917754606: {'path': 'temp/1758688614_828627_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou step Faster rcnn !
classes : ['background', 'plaque']
pht : 4370
caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'}
To loadFromThcl()
model_param file didn't exist 
model_name : detection_plaque_valcor_010622
model_type : caffe_faster_rcnn
list file need :
['caffemodel', 'test.prototxt']
file exist in s3 :
['caffemodel', 'test.prototxt']
file manque in s3 : 
[]
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0924 06:36:56.479066 828627 common.cpp:114] Cannot create Cublas handle. Cublas won't be available.
E0924 06:36:56.485709 828627 common.cpp:121] Cannot create Curand generator. Curand won't be available.
F0924 06:36:56.511749 828627 syncedmem.hpp:22] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
Command terminated by signal 6
6.05user 6.84system 1:31.35elapsed 14%CPU (0avgtext+0avgdata 1099680maxresident)k
2378864inputs+3368outputs (8004major+572566minor)pagefaults 0swaps