python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120
 import MySQLdb succeeded 
Import error (python version)
python version = 3
warning , we can't find thcl infos in json_data
warning , we can't find pdt infos in json_data
 list_job_run_as_list :  ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel']
python version used : 3
 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')]


#&_# BEGIN OF TEST : tests/mask_test #&_# 


/home/admin/workarea/git/Velours/python/tests/mask_test.py
<module 'tests.mask_test' from '/home/admin/workarea/git/Velours/python/tests/mask_test.py'>
Test mask-detection 
python version used : 3

############################### TEST memory used  ################################
free memory at begining : 
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 205

run mask_detect

Inside batchDatouExec : verbose : False
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
List Step Type Loaded in datou : mask_detect
list_input_json : []
origin
BFwe have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.6980350017547607
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Fri Feb 28 06:35:28 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3000
l 3632 free memory gpu now : 205
wait 20 seconds
l 3637 free memory gpu now : 205  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
/home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
2025-02-28 06:35:50.761799: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-02-28 06:35:50.787152: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz
2025-02-28 06:35:50.789278: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f07b4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-02-28 06:35:50.789359: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-02-28 06:35:50.793398: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-02-28 06:35:51.038843: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2bd31c00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-02-28 06:35:51.038892: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-02-28 06:35:51.039644: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 06:35:51.039988: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 06:35:51.042632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 06:35:51.045309: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 06:35:51.045722: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 06:35:51.048627: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 06:35:51.050024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 06:35:51.055532: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 06:35:51.056589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 06:35:51.056674: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 06:35:51.057260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-02-28 06:35:51.057281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-02-28 06:35:51.057293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-02-28 06:35:51.058230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 53 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
To do loadFromThcl(), then load ParamDescType : thcl454
 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3473
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
2025-02-28 06:35:51.594432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 06:35:51.594523: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 06:35:51.594542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 06:35:51.594557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 06:35:51.594573: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 06:35:51.594587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 06:35:51.594613: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 06:35:51.594628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 06:35:51.595329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 06:35:51.596210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 06:35:51.596248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 06:35:51.596267: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 06:35:51.596285: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 06:35:51.596302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 06:35:51.596319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 06:35:51.596336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 06:35:51.596354: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 06:35:51.597172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 06:35:51.597205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-02-28 06:35:51.597216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-02-28 06:35:51.597226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-02-28 06:35:51.598064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 53 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
2025-02-28 06:35:51.797152: F ./tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: out of memory
max_time_sub_proc : 3600

Useless call to update_current_state
in case -12
caffe_path_current : 
About to save ! 1
Inside saveOutput : final : True verbose : False
eke 12-6-18 : saveMask need to be cleaned for new output !
ERROR : mask output needs to be a dictionnary now !
No output to save, continue without doing anything !

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : -12
free memory after detection : 
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 4284

############################### TEST detect object  ################################

run mask_detect

Inside batchDatouExec : verbose : False
Catched exception !
Connect or reconnect !
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
List Step Type Loaded in datou : mask_detect
list_input_json : []
origin
BFwe have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.23475241661071777
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Fri Feb 28 07:35:50 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
l 3637 free memory gpu now : 4284  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
2025-02-28 07:35:54.838692: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-02-28 07:35:54.867166: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz
2025-02-28 07:35:54.868759: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f07b4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-02-28 07:35:54.868788: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-02-28 07:35:54.872325: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-02-28 07:35:55.032381: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2bb83640 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-02-28 07:35:55.032443: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-02-28 07:35:55.033405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 07:35:55.033898: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 07:35:55.037073: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 07:35:55.040246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 07:35:55.040800: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 07:35:55.044329: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 07:35:55.045396: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 07:35:55.049852: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 07:35:55.050991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 07:35:55.051085: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 07:35:55.051686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-02-28 07:35:55.051722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-02-28 07:35:55.051733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-02-28 07:35:55.052733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3832 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
To do loadFromThcl(), then load ParamDescType : thcl454
 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3473
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
2025-02-28 07:35:55.729626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 07:35:55.729738: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 07:35:55.729762: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 07:35:55.729784: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 07:35:55.729805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 07:35:55.729825: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 07:35:55.729846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 07:35:55.729867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 07:35:55.731061: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 07:35:55.732687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 07:35:55.732813: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 07:35:55.732845: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 07:35:55.732871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 07:35:55.732897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 07:35:55.732926: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 07:35:55.732952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 07:35:55.732979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 07:35:55.734448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 07:35:55.734522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-02-28 07:35:55.734539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-02-28 07:35:55.734552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-02-28 07:35:55.736263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3832 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
Using TensorFlow backend.
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


model_param file didn't exist 
model_name : mask_coco_origin
model_type : mask_rcnn
list file need :
['mask_model.h5']
file exist in s3 :
['mask_model.h5']
file manque in s3 : 
[]
2025-02-28 07:36:03.966792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 07:36:04.292560: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 07:36:06.247003: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:06.247765: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.92G (3134914560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:06.248385: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.63G (2821423104 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.032627: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2025-02-28 07:36:07.095300: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.095422: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.096793: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.096841: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.104875: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.104947: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.111938: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.111999: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.112702: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.112727: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.139516: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.139632: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 19.91MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.139652: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2025-02-28 07:36:07.140840: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.140868: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.142034: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.142063: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.149763: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.149821: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-02-28 07:36:07.150987: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.152196: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.153354: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.162986: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.164255: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.182705: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.183719: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.184542: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.185220: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.190090: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.190841: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.191568: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.192275: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.194497: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.204390: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.205170: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.216898: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.217723: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.218423: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.219097: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.219783: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:07.220448: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3483238400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
local folder  : /data/models_weight/mask_coco_origin


/data/models_weight/mask_coco_origin/mask_model.h5
size_local : 257557808 size in s3 : 257557808
create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17
mask_model.h5 already exist and didn't need to update
 list_images length : 1
NEW PHOTO
Processing 1 images
image                    shape: (720, 1280, 3)        min:    0.00000  max:  255.00000
molded_images            shape: (1, 640, 640, 3)      min: -123.70000  max:  151.10000
image_metas              shape: (1, 89)               min:    0.00000  max: 1280.00000
nb d'objets trouves : 4
Detection mask done !
Trying to reset tf kernel
3608269
begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 2697
tf kernel not reseted
sub process len(results) : 1 len(list_Values) 0
None
max_time_sub_proc : 3600
parent process len(results) : 1 len(list_Values) 0
process is alive
finish correctly or not : True

 after detect 

begin to check gpu status 
inside check gpu memory
l 3610 free memory gpu now : 3890
list_Values should be empty []
To do loadFromThcl(), then load ParamDescType : thcl454
Catched exception !
Connect or reconnect !
 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3473
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']
time for calcul the mask position with numpy : 0.0004169940948486328
nb_pixel_total : 16902
time to create 1 rle with old method : 0.01920795440673828
length of segment :  107
time for calcul the mask position with numpy : 0.06484818458557129
nb_pixel_total : 480754
time to create 1 rle with new method : 0.03780865669250488
length of segment :  632
time for calcul the mask position with numpy : 0.0008113384246826172
nb_pixel_total : 36640
time to create 1 rle with old method : 0.044063568115234375
length of segment :  133
time for calcul the mask position with numpy : 0.00010657310485839844
nb_pixel_total : 4791
time to create 1 rle with old method : 0.006670713424682617
length of segment :  51

time spent for convertir_results : 1.1138379573822021

time spend for datou_step_exec : 20.90599536895752
time spend to save output : 6.365776062011719e-05
total time spend for step 1 : 20.90605902671814

caffe_path_current : 
About to save ! 1
Inside saveOutput : final : True verbose : False
eke 12-6-18 : saveMask need to be cleaned for new output !
Number saved : None
batch 1
Loaded 400 chid ids of type : 445
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 1
time used for this insertion : 0.01276087760925293

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : {'917855882': [[(917855882, 492601069, 445, 1092, 1280, 0, 108, 0.9988368, [(1205, 1, 58), (1165, 2, 105), (1159, 3, 113), (1149, 4, 124), (1113, 5, 161), (1100, 6, 174), (1097, 7, 177), (1095, 8, 179), (1095, 9, 179), (1095, 10, 179), (1095, 11, 179), (1095, 12, 179), (1095, 13, 179), (1095, 14, 178), (1095, 15, 178), (1095, 16, 178), (1095, 17, 178), (1095, 18, 177), (1095, 19, 177), (1095, 20, 177), (1095, 21, 177), (1095, 22, 177), (1095, 23, 178), (1095, 24, 178), (1095, 25, 178), (1095, 26, 179), (1095, 27, 179), (1095, 28, 180), (1095, 29, 181), (1095, 30, 182), (1095, 31, 183), (1095, 32, 183), (1095, 33, 184), (1095, 34, 184), (1096, 35, 183), (1096, 36, 183), (1096, 37, 184), (1097, 38, 183), (1097, 39, 183), (1097, 40, 183), (1098, 41, 182), (1098, 42, 182), (1098, 43, 182), (1099, 44, 181), (1099, 45, 181), (1099, 46, 181), (1100, 47, 180), (1100, 48, 180), (1101, 49, 179), (1101, 50, 179), (1102, 51, 178), (1102, 52, 178), (1103, 53, 177), (1103, 54, 177), (1104, 55, 176), (1104, 56, 176), (1104, 57, 176), (1104, 58, 176), (1105, 59, 175), (1105, 60, 175), (1105, 61, 175), (1105, 62, 175), (1105, 63, 175), (1106, 64, 174), (1106, 65, 174), (1106, 66, 174), (1106, 67, 174), (1106, 68, 174), (1106, 69, 174), (1106, 70, 174), (1106, 71, 174), (1106, 72, 174), (1106, 73, 174), (1107, 74, 173), (1107, 75, 173), (1107, 76, 173), (1107, 77, 173), (1107, 78, 173), (1107, 79, 173), (1108, 80, 172), (1108, 81, 172), (1109, 82, 171), (1110, 83, 170), (1110, 84, 170), (1111, 85, 169), (1112, 86, 168), (1113, 87, 166), (1114, 88, 165), (1115, 89, 164), (1117, 90, 162), (1120, 91, 159), (1138, 92, 141), (1146, 93, 133), (1154, 94, 125), (1167, 95, 112), (1177, 96, 102), (1183, 97, 95), (1185, 98, 93), (1187, 99, 90), (1188, 100, 55), (1264, 100, 12), (1190, 101, 50), (1191, 102, 46), (1194, 103, 40), (1197, 104, 34), (1202, 105, 25), (1207, 106, 16)], ['1222,106,1207,106,1206,105,1197,104,1191,102,1182,96,1176,95,1167,95,1166,94,1154,94,1153,93,1146,93,1145,92,1137,91,1120,91,1115,89,1110,84,1107,79,1106,73,1106,64,1104,55,1099,46,1095,34,1095,8,1100,6,1112,6,1113,5,1148,5,1149,4,1158,4,1165,2,1204,2,1205,1,1262,1,1269,2,1273,5,1273,13,1271,18,1271,22,1273,27,1277,31,1279,37,1279,86,1278,87,1278,96,1275,100,1264,100,1263,99,1243,99,1230,104']), (917855882, 492601069, 445, 52, 1128, 16, 668, 0.99774605, [(710, 22, 23), (925, 22, 47), (608, 23, 146), (893, 23, 104), (598, 24, 234), (849, 24, 159), (589, 25, 428), (582, 26, 444), (574, 27, 459), (569, 28, 466), (565, 29, 472), (560, 30, 480), (555, 31, 487), (550, 32, 495), (544, 33, 503), (538, 34, 512), (532, 35, 520), (527, 36, 527), (523, 37, 534), (518, 38, 541), (514, 39, 548), (510, 40, 554), (506, 41, 561), (503, 42, 566), (499, 43, 572), (496, 44, 577), (493, 45, 582), (491, 46, 585), (489, 47, 589), (487, 48, 592), (485, 49, 595), (483, 50, 598), (482, 51, 600), (481, 52, 602), (480, 53, 603), (479, 54, 605), (478, 55, 606), (477, 56, 607), (475, 57, 610), (474, 58, 611), (473, 59, 613), (472, 60, 614), (470, 61, 616), (469, 62, 618), (468, 63, 619), (466, 64, 621), (465, 65, 623), (464, 66, 624), (462, 67, 626), (461, 68, 628), (459, 69, 630), (458, 70, 631), (456, 71, 633), (455, 72, 635), (453, 73, 637), (452, 74, 638), (451, 75, 639), (450, 76, 640), (448, 77, 642), (447, 78, 643), (446, 79, 644), (445, 80, 645), (444, 81, 646), (442, 82, 648), (441, 83, 649), (440, 84, 650), (439, 85, 651), (438, 86, 652), (437, 87, 653), (436, 88, 654), (435, 89, 655), (434, 90, 656), (433, 91, 657), (432, 92, 658), (431, 93, 659), (430, 94, 660), (429, 95, 661), (428, 96, 662), (427, 97, 663), (425, 98, 665), (423, 99, 667), (421, 100, 669), (419, 101, 671), (417, 102, 673), (413, 103, 677), (410, 104, 680), (405, 105, 685), (401, 106, 689), (397, 107, 693), (392, 108, 698), (387, 109, 703), (382, 110, 708), (377, 111, 713), (373, 112, 717), (369, 113, 721), (365, 114, 725), (362, 115, 728), (358, 116, 732), (356, 117, 734), (353, 118, 737), (351, 119, 739), (349, 120, 741), (346, 121, 744), (344, 122, 746), (341, 123, 749), (338, 124, 752), (335, 125, 755), (331, 126, 759), (327, 127, 763), (323, 128, 767), (319, 129, 770), (314, 130, 775), (308, 131, 781), (303, 132, 786), (294, 133, 795), (287, 134, 802), (279, 135, 810), (273, 136, 816), (267, 137, 822), (262, 138, 827), (258, 139, 831), (255, 140, 834), (252, 141, 837), (250, 142, 839), (247, 143, 842), (245, 144, 844), (242, 145, 847), (240, 146, 849), (237, 147, 852), (234, 148, 855), (230, 149, 859), (226, 150, 863), (220, 151, 869), (213, 152, 876), (207, 153, 882), (200, 154, 889), (193, 155, 896), (187, 156, 902), (183, 157, 906), (181, 158, 908), (178, 159, 911), (176, 160, 913), (174, 161, 915), (172, 162, 917), (170, 163, 919), (168, 164, 921), (166, 165, 923), (165, 166, 924), (164, 167, 925), (162, 168, 927), (161, 169, 928), (159, 170, 930), (157, 171, 932), (155, 172, 934), (153, 173, 935), (151, 174, 937), (148, 175, 940), (146, 176, 942), (144, 177, 944), (142, 178, 946), (140, 179, 948), (139, 180, 949), (137, 181, 951), (136, 182, 952), (134, 183, 954), (133, 184, 955), (132, 185, 956), (131, 186, 957), (130, 187, 958), (129, 188, 959), (128, 189, 960), (127, 190, 960), (126, 191, 961), (126, 192, 961), (125, 193, 962), (124, 194, 963), (123, 195, 964), (122, 196, 965), (122, 197, 965), (121, 198, 966), (120, 199, 967), (119, 200, 968), (118, 201, 969), (117, 202, 970), (116, 203, 971), (114, 204, 973), (113, 205, 973), (112, 206, 974), (111, 207, 975), (109, 208, 977), (108, 209, 978), (107, 210, 979), (106, 211, 980), (105, 212, 981), (104, 213, 982), (103, 214, 983), (102, 215, 984), (101, 216, 985), (101, 217, 984), (100, 218, 985), (100, 219, 985), (99, 220, 986), (98, 221, 987), (98, 222, 987), (97, 223, 988), (97, 224, 987), (96, 225, 988), (96, 226, 988), (95, 227, 989), (95, 228, 989), (94, 229, 990), (94, 230, 990), (94, 231, 990), (93, 232, 990), (93, 233, 990), (92, 234, 991), (92, 235, 991), (92, 236, 991), (91, 237, 992), (91, 238, 991), (91, 239, 991), (91, 240, 990), (91, 241, 990), (90, 242, 991), (90, 243, 990), (90, 244, 990), (90, 245, 989), (90, 246, 989), (89, 247, 990), (89, 248, 989), (89, 249, 989), (89, 250, 988), (89, 251, 988), (88, 252, 988), (88, 253, 988), (88, 254, 987), (88, 255, 986), (88, 256, 986), (87, 257, 986), (87, 258, 985), (87, 259, 985), (87, 260, 984), (87, 261, 983), (86, 262, 983), (86, 263, 982), (86, 264, 982), (86, 265, 981), (85, 266, 981), (85, 267, 980), (85, 268, 980), (85, 269, 979), (84, 270, 979), (84, 271, 979), (84, 272, 978), (83, 273, 979), (83, 274, 978), (83, 275, 977), (82, 276, 978), (82, 277, 977), (82, 278, 977), (81, 279, 977), (81, 280, 977), (81, 281, 977), (80, 282, 977), (80, 283, 977), (80, 284, 976), (79, 285, 977), (79, 286, 976), (79, 287, 976), (78, 288, 976), (78, 289, 976), (78, 290, 975), (77, 291, 976), (77, 292, 975), (77, 293, 975), (76, 294, 975), (76, 295, 975), (76, 296, 974), (75, 297, 975), (75, 298, 974), (74, 299, 975), (74, 300, 974), (74, 301, 974), (73, 302, 974), (73, 303, 974), (72, 304, 974), (72, 305, 974), (71, 306, 974), (71, 307, 973), (71, 308, 972), (70, 309, 972), (70, 310, 971), (70, 311, 970), (70, 312, 968), (69, 313, 968), (69, 314, 966), (69, 315, 964), (69, 316, 962), (68, 317, 961), (68, 318, 959), (68, 319, 958), (68, 320, 956), (67, 321, 955), (67, 322, 954), (67, 323, 952), (67, 324, 951), (66, 325, 951), (66, 326, 950), (66, 327, 948), (66, 328, 947), (65, 329, 947), (65, 330, 946), (65, 331, 946), (65, 332, 945), (65, 333, 944), (65, 334, 942), (65, 335, 941), (65, 336, 940), (65, 337, 939), (65, 338, 938), (65, 339, 936), (64, 340, 936), (64, 341, 934), (64, 342, 932), (64, 343, 930), (64, 344, 928), (64, 345, 927), (64, 346, 925), (64, 347, 923), (64, 348, 922), (64, 349, 920), (64, 350, 919), (63, 351, 919), (63, 352, 918), (63, 353, 917), (63, 354, 916), (63, 355, 915), (63, 356, 914), (63, 357, 912), (63, 358, 911), (63, 359, 910), (63, 360, 909), (63, 361, 908), (63, 362, 906), (63, 363, 905), (63, 364, 904), (63, 365, 902), (63, 366, 901), (63, 367, 899), (63, 368, 898), (62, 369, 897), (62, 370, 895), (62, 371, 893), (62, 372, 891), (62, 373, 890), (62, 374, 888), (62, 375, 887), (62, 376, 886), (62, 377, 885), (62, 378, 884), (62, 379, 883), (63, 380, 880), (63, 381, 879), (63, 382, 878), (63, 383, 877), (63, 384, 876), (63, 385, 875), (63, 386, 874), (63, 387, 873), (63, 388, 872), (64, 389, 870), (64, 390, 869), (64, 391, 868), (64, 392, 867), (64, 393, 866), (64, 394, 864), (64, 395, 863), (65, 396, 861), (65, 397, 860), (65, 398, 859), (65, 399, 858), (65, 400, 857), (65, 401, 856), (65, 402, 854), (65, 403, 853), (65, 404, 851), (65, 405, 850), (65, 406, 848), (66, 407, 846), (66, 408, 844), (66, 409, 843), (66, 410, 842), (66, 411, 841), (66, 412, 840), (66, 413, 838), (66, 414, 837), (66, 415, 836), (66, 416, 835), (66, 417, 835), (66, 418, 834), (66, 419, 833), (66, 420, 832), (67, 421, 830), (67, 422, 829), (67, 423, 829), (67, 424, 828), (67, 425, 827), (67, 426, 826), (67, 427, 825), (67, 428, 824), (67, 429, 823), (68, 430, 820), (68, 431, 819), (68, 432, 818), (68, 433, 816), (68, 434, 815), (68, 435, 813), (68, 436, 811), (69, 437, 809), (69, 438, 807), (69, 439, 806), (69, 440, 804), (69, 441, 803), (69, 442, 802), (69, 443, 800), (70, 444, 798), (70, 445, 797), (70, 446, 796), (70, 447, 796), (71, 448, 794), (71, 449, 794), (72, 450, 792), (72, 451, 792), (73, 452, 790), (73, 453, 789), (74, 454, 788), (74, 455, 787), (75, 456, 786), (75, 457, 785), (76, 458, 784), (76, 459, 783), (77, 460, 782), (77, 461, 781), (77, 462, 781), (78, 463, 779), (78, 464, 779), (79, 465, 777), (79, 466, 777), (79, 467, 776), (80, 468, 775), (80, 469, 774), (80, 470, 774), (81, 471, 772), (81, 472, 771), (82, 473, 770), (82, 474, 769), (83, 475, 767), (83, 476, 766), (83, 477, 766), (84, 478, 764), (84, 479, 763), (85, 480, 761), (85, 481, 760), (85, 482, 759), (86, 483, 757), (86, 484, 755), (87, 485, 753), (87, 486, 752), (87, 487, 751), (88, 488, 748), (88, 489, 747), (88, 490, 746), (89, 491, 744), (89, 492, 743), (90, 493, 741), (90, 494, 741), (91, 495, 739), (91, 496, 738), (92, 497, 737), (93, 498, 735), (94, 499, 733), (94, 500, 733), (95, 501, 731), (96, 502, 729), (97, 503, 728), (98, 504, 726), (99, 505, 724), (99, 506, 724), (100, 507, 722), (101, 508, 721), (102, 509, 719), (104, 510, 717), (105, 511, 715), (106, 512, 714), (107, 513, 712), (108, 514, 711), (110, 515, 708), (111, 516, 707), (113, 517, 704), (114, 518, 703), (115, 519, 701), (117, 520, 698), (118, 521, 697), (119, 522, 695), (121, 523, 693), (122, 524, 691), (124, 525, 689), (125, 526, 687), (126, 527, 685), (128, 528, 683), (129, 529, 681), (131, 530, 678), (132, 531, 676), (134, 532, 674), (135, 533, 672), (137, 534, 669), (138, 535, 667), (140, 536, 664), (141, 537, 662), (143, 538, 659), (144, 539, 657), (146, 540, 654), (148, 541, 651), (149, 542, 649), (151, 543, 645), (153, 544, 642), (154, 545, 641), (156, 546, 638), (158, 547, 635), (159, 548, 633), (161, 549, 630), (162, 550, 628), (164, 551, 626), (166, 552, 623), (167, 553, 621), (169, 554, 618), (170, 555, 617), (171, 556, 615), (173, 557, 613), (174, 558, 611), (176, 559, 608), (177, 560, 607), (179, 561, 604), (180, 562, 603), (181, 563, 601), (183, 564, 599), (185, 565, 597), (186, 566, 595), (189, 567, 592), (192, 568, 589), (195, 569, 585), (198, 570, 582), (201, 571, 579), (204, 572, 575), (206, 573, 573), (209, 574, 569), (212, 575, 566), (215, 576, 563), (218, 577, 559), (221, 578, 556), (224, 579, 552), (226, 580, 550), (228, 581, 547), (230, 582, 545), (232, 583, 542), (234, 584, 540), (235, 585, 539), (237, 586, 536), (238, 587, 534), (240, 588, 531), (242, 589, 528), (243, 590, 526), (245, 591, 523), (247, 592, 520), (249, 593, 516), (251, 594, 513), (253, 595, 510), (256, 596, 505), (258, 597, 501), (261, 598, 497), (263, 599, 493), (267, 600, 488), (271, 601, 482), (274, 602, 478), (278, 603, 473), (281, 604, 468), (284, 605, 464), (287, 606, 460), (290, 607, 456), (292, 608, 453), (295, 609, 449), (297, 610, 446), (300, 611, 442), (303, 612, 438), (305, 613, 434), (307, 614, 431), (310, 615, 427), (312, 616, 423), (315, 617, 418), (317, 618, 415), (320, 619, 410), (322, 620, 406), (325, 621, 401), (327, 622, 396), (330, 623, 390), (333, 624, 384), (335, 625, 379), (338, 626, 374), (341, 627, 369), (345, 628, 362), (349, 629, 356), (353, 630, 350), (357, 631, 344), (360, 632, 340), (364, 633, 334), (368, 634, 328), (373, 635, 320), (378, 636, 313), (383, 637, 305), (389, 638, 295), (395, 639, 282), (401, 640, 270), (408, 641, 256), (416, 642, 240), (431, 643, 217), (448, 644, 193), (465, 645, 169), (480, 646, 148), (495, 647, 126), (511, 648, 104), (526, 649, 81), (565, 650, 9)], ['526,649,416,642,341,627,289,606,263,599,223,578,186,566,102,509,91,496,70,447,62,379,65,329,85,269,91,237,101,216,134,183,187,156,225,151,279,135,343,123,358,116,416,103,426,98,483,50,506,41,608,23,754,24,892,24,925,22,996,23,1032,27,1066,41,1082,52,1089,72,1088,172,1082,237,1045,305,1019,322,1002,338,950,373,895,423,865,446,851,473,822,505,810,528,786,554,773,585,714,624,683,638,606,649']), (917855882, 492601069, 445, 0, 440, 0, 116, 0.9919383, [(127, 1, 141), (95, 2, 205), (384, 2, 2), (59, 3, 273), (340, 3, 57), (22, 4, 381), (19, 5, 387), (16, 6, 392), (15, 7, 394), (14, 8, 396), (14, 9, 397), (13, 10, 399), (12, 11, 400), (12, 12, 400), (11, 13, 402), (10, 14, 403), (11, 15, 403), (11, 16, 404), (12, 17, 403), (12, 18, 404), (12, 19, 405), (12, 20, 405), (12, 21, 406), (12, 22, 406), (12, 23, 407), (12, 24, 407), (12, 25, 408), (12, 26, 408), (12, 27, 408), (12, 28, 408), (12, 29, 409), (12, 30, 409), (12, 31, 409), (12, 32, 409), (12, 33, 409), (12, 34, 410), (12, 35, 410), (12, 36, 410), (12, 37, 410), (12, 38, 410), (12, 39, 410), (12, 40, 410), (12, 41, 411), (12, 42, 411), (12, 43, 411), (12, 44, 411), (12, 45, 411), (12, 46, 410), (12, 47, 410), (12, 48, 410), (12, 49, 410), (12, 50, 410), (12, 51, 410), (12, 52, 409), (12, 53, 408), (12, 54, 408), (12, 55, 407), (12, 56, 406), (12, 57, 404), (12, 58, 403), (11, 59, 403), (11, 60, 402), (11, 61, 401), (11, 62, 400), (11, 63, 400), (11, 64, 399), (11, 65, 398), (11, 66, 397), (11, 67, 397), (11, 68, 396), (11, 69, 395), (11, 70, 395), (11, 71, 394), (11, 72, 394), (11, 73, 394), (11, 74, 393), (11, 75, 393), (11, 76, 393), (11, 77, 393), (11, 78, 393), (11, 79, 393), (11, 80, 392), (10, 81, 394), (10, 82, 394), (10, 83, 395), (9, 84, 396), (9, 85, 262), (279, 85, 126), (9, 86, 75), (98, 86, 28), (142, 86, 117), (292, 86, 112), (9, 87, 71), (152, 87, 103), (294, 87, 110), (8, 88, 68), (161, 88, 91), (296, 88, 107), (8, 89, 63), (177, 89, 72), (297, 89, 106), (7, 90, 61), (205, 90, 40), (298, 90, 104), (7, 91, 57), (299, 91, 103), (6, 92, 54), (300, 92, 102), (6, 93, 50), (301, 93, 100), (7, 94, 46), (303, 94, 97), (7, 95, 44), (306, 95, 92), (7, 96, 42), (308, 96, 89), (7, 97, 40), (310, 97, 86), (7, 98, 38), (312, 98, 83), (8, 99, 34), (314, 99, 79), (8, 100, 32), (317, 100, 75), (8, 101, 29), (319, 101, 71), (13, 102, 19), (324, 102, 63), (20, 103, 6), (330, 103, 51), (337, 104, 37), (344, 105, 22), (352, 106, 3)], ['344,105,319,101,301,93,291,85,259,85,244,90,205,90,204,89,177,89,176,88,161,88,160,87,142,86,141,85,126,85,125,86,98,86,84,85,56,92,36,101,26,102,8,101,6,92,11,80,11,59,12,58,12,17,10,14,16,6,22,4,94,3,95,2,126,2,127,1,267,1,268,2,396,3,407,6,419,25,421,34,421,51,410,62,404,71,402,80,404,85,401,92,394,98,386,102,374,103,365,105']), (917855882, 492601069, 445, 390, 550, 0, 54, 0.9390803, [(415, 0, 5), (441, 0, 60), (508, 0, 28), (402, 1, 142), (401, 2, 146), (402, 3, 145), (404, 4, 143), (406, 5, 140), (408, 6, 137), (410, 7, 134), (411, 8, 132), (412, 9, 130), (413, 10, 127), (414, 11, 125), (415, 12, 123), (415, 13, 122), (416, 14, 120), (417, 15, 117), (417, 16, 116), (418, 17, 114), (418, 18, 113), (418, 19, 111), (418, 20, 109), (419, 21, 107), (419, 22, 105), (419, 23, 103), (419, 24, 102), (419, 25, 100), (420, 26, 97), (420, 27, 95), (420, 28, 94), (421, 29, 91), (421, 30, 90), (422, 31, 88), (422, 32, 88), (422, 33, 87), (423, 34, 84), (423, 35, 82), (423, 36, 81), (424, 37, 79), (424, 38, 77), (424, 39, 75), (424, 40, 73), (424, 41, 71), (425, 42, 67), (425, 43, 66), (426, 44, 62), (426, 45, 6), (433, 45, 52), (443, 46, 29), (450, 47, 1)], ['450,47,449,46,443,46,442,45,426,45,424,41,424,37,423,36,422,31,419,25,419,21,418,20,418,17,417,15,409,6,402,3,402,1,414,1,415,0,419,0,420,1,440,1,441,0,500,0,501,1,507,1,508,0,535,0,536,1,543,1,546,2,546,4,542,8,530,18,527,19,525,21,522,22,520,24,512,28,508,33,505,34,502,37,494,41,492,41,490,43,488,43,484,45,472,45,471,46,451,46'])], 'temp/1740724550_3374456_917855882_da0fa7b7e6b5b551fe26c0ba8713276d.jpg']}

############################### TEST POLYGON  ################################
Inside batchDatouExec : verbose : False
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
List Step Type Loaded in datou : mask_detect
list_input_json : []
origin
BFwe have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
time to download the photos : 0.17711210250854492
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : False
number of steps :  1
step1:mask_detect
Fri Feb 28 07:36:18 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
Beginning of datou step mask_detect !
save_polygon : True

 begin detect 

begin to check gpu status 
inside check gpu memory
l 3637 free memory gpu now : 3890  max_wait_temp :  1 max_wait :  0
 gpu_flag :  0
2025-02-28 07:36:21.594418: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-02-28 07:36:21.630335: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz
2025-02-28 07:36:21.632307: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f07b4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-02-28 07:36:21.632343: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-02-28 07:36:21.636574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-02-28 07:36:21.816663: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2c99fb50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-02-28 07:36:21.816770: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-02-28 07:36:21.818253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 07:36:21.818913: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 07:36:21.822187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 07:36:21.825858: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 07:36:21.826302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 07:36:21.830754: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 07:36:21.832164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 07:36:21.837308: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 07:36:21.838695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 07:36:21.838839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 07:36:21.839525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-02-28 07:36:21.839546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-02-28 07:36:21.839556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-02-28 07:36:21.840581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3438 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

2025-02-28 07:36:22.167006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 07:36:22.167200: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 07:36:22.167230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 07:36:22.167253: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 07:36:22.167274: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 07:36:22.167295: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 07:36:22.167316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 07:36:22.167338: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 07:36:22.168439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 07:36:22.169809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 07:36:22.169866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 07:36:22.169888: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 07:36:22.169909: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 07:36:22.169933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 07:36:22.169953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 07:36:22.169972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 07:36:22.169992: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 07:36:22.171053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 07:36:22.171118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-02-28 07:36:22.171131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-02-28 07:36:22.171141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-02-28 07:36:22.172241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3438 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
Using TensorFlow backend.
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Inside mask_sub_process
Inside mask_detect 
About to load cache.load_thcl_param 
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21))
{'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0}
list_class_names : 
['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush']

Configurations:
BACKBONE                       resnet101
BACKBONE_SHAPES                [[160 160]
 [ 80  80]
 [ 40  40]
 [ 20  20]
 [ 10  10]]
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.3
DETECTION_NMS_THRESHOLD        0.3
GPU_COUNT                      1
IMAGES_PER_GPU                 1
IMAGE_MAX_DIM                  640
IMAGE_MIN_DIM                  640
IMAGE_PADDING                  True
IMAGE_SHAPE                    [640 640   3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           mask_coco_origin
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (16, 32, 64, 128, 256)
RPN_ANCHOR_STRIDE              1
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


model_param file didn't exist 
model_name : mask_coco_origin
model_type : mask_rcnn
list file need :
['mask_model.h5']
file exist in s3 :
['mask_model.h5']
file manque in s3 : 
[]
2025-02-28 07:36:27.807468: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 51380224 exceeds 10% of free system memory.
2025-02-28 07:36:35.153333: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 07:36:35.502395: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 07:36:37.255108: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3070099456 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:37.255822: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.57G (2763089408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 07:36:38.058995: E tensorflow/stream_executor/cuda/cuda_driver.cc:910] failed to synchronize the stop event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2025-02-28 07:36:38.059438: E tensorflow/stream_executor/gpu/gpu_timer.cc:55] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2025-02-28 07:36:38.059453: E tensorflow/stream_executor/gpu/gpu_timer.cc:60] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2025-02-28 07:36:38.059512: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2025-02-28 07:36:38.059529: E tensorflow/stream_executor/stream.cc:5485] Internal: Failed to enqueue async memset operation: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2025-02-28 07:36:38.059556: W tensorflow/core/kernels/gpu_utils.cc:69] Failed to check cudnn convolutions for out-of-bounds reads and writes with an error message: 'Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered'; skipping this check. This only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2025-02-28 07:36:38.059567: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2025-02-28 07:36:38.059580: I tensorflow/stream_executor/stream.cc:4963] [stream=0x2db3bc30,impl=0x2db3ac20] did not memzero GPU location; source: 0x7f05fb7fc020
2025-02-28 07:36:38.060199: F ./tensorflow/core/kernels/reduction_gpu_kernels.cu.h:731] Non-OK-status: GpuLaunchKernel(RowReduceKernel<IN_T, OUT_T, Op>, num_blocks, threads_per_block, 0, cu_stream, in, out, num_rows, num_cols, op, init) status: Internal: an illegal memory access was encountered
max_time_sub_proc : 3600

Useless call to update_current_state
in case -12
caffe_path_current : 
About to save ! 1
Inside saveOutput : final : True verbose : False
eke 12-6-18 : saveMask need to be cleaned for new output !
ERROR : mask output needs to be a dictionnary now !
No output to save, continue without doing anything !

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : -12
ERROR : 'int' object is not subscriptable
reconnect to base !
warning , we can't find thcl infos in json_data
warning , we can't find pdt infos in json_data


#&_# TEST FAILED #&_# : tests/mask_test #&_# 


Error : invalid literal for int() with base 10: "'int' object is not subscriptable"
/home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_3bb8fc9eb89f6e73213a93d2f1429765ec1e113a
SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_3bb8fc9eb89f6e73213a93d2f1429765ec1e113a','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/February/28022025/python_test3//data_2/data_log/job/2025/February/28022025/python_test3/log-python3----short_python3--v--marlene-06:35:00.txt','mask_detection','unknown');

#&_# END OF TEST #&_# : tests/mask_test #&_# 





#&_# BEGIN OF TEST : tests/datou_test #&_# 


/home/admin/workarea/git/Velours/python/tests/datou_test.py
<module 'tests.datou_test' from '/home/admin/workarea/git/Velours/python/tests/datou_test.py'>
Datou All Test
python version used : 3

############################### TEST sam  ################################
TEST SAM
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4573
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4573
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4573
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : sam
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (1189321094) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 1189321094
download finish for photo 1189321094
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.3231959342956543
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
WARNING : we have an input that is not a photo, we should get rid of it
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:sam
Fri Feb 28 08:36:22 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1740728182_3374456_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094}
map_photo_id_path_extension : {1189321094: {'path': 'temp/1740728182_3374456_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}}
map_subphoto_mainphoto : {}

Beginning of datou step sam !
pht : 4677
Inside sam : nb paths : 1
(640, 960, 3)
time for calcul the mask position with numpy : 0.0023016929626464844
nb_pixel_total : 2940
time to create 1 rle with old method : 0.003803253173828125
time for calcul the mask position with numpy : 0.0015568733215332031
nb_pixel_total : 10791
time to create 1 rle with old method : 0.012521982192993164
time for calcul the mask position with numpy : 0.0015039443969726562
nb_pixel_total : 1219
time to create 1 rle with old method : 0.0015273094177246094
time for calcul the mask position with numpy : 0.001501321792602539
nb_pixel_total : 8649
time to create 1 rle with old method : 0.010026931762695312
time for calcul the mask position with numpy : 0.002462625503540039
nb_pixel_total : 83847
time to create 1 rle with old method : 0.095489501953125
time for calcul the mask position with numpy : 0.0015537738800048828
nb_pixel_total : 5623
time to create 1 rle with old method : 0.006747007369995117
time for calcul the mask position with numpy : 0.0014922618865966797
nb_pixel_total : 5493
time to create 1 rle with old method : 0.0066792964935302734
time for calcul the mask position with numpy : 0.0016751289367675781
nb_pixel_total : 6630
time to create 1 rle with old method : 0.007871150970458984
time for calcul the mask position with numpy : 0.0015249252319335938
nb_pixel_total : 13925
time to create 1 rle with old method : 0.01619744300842285
time for calcul the mask position with numpy : 0.0014581680297851562
nb_pixel_total : 3780
time to create 1 rle with old method : 0.00439906120300293
time for calcul the mask position with numpy : 0.0014998912811279297
nb_pixel_total : 9896
time to create 1 rle with old method : 0.011721372604370117
time for calcul the mask position with numpy : 0.0017418861389160156
nb_pixel_total : 16321
time to create 1 rle with old method : 0.01894402503967285
time for calcul the mask position with numpy : 0.0017638206481933594
nb_pixel_total : 29433
time to create 1 rle with old method : 0.04496502876281738
time for calcul the mask position with numpy : 0.0016736984252929688
nb_pixel_total : 1820
time to create 1 rle with old method : 0.0032885074615478516
time for calcul the mask position with numpy : 0.0016200542449951172
nb_pixel_total : 4281
time to create 1 rle with old method : 0.007296323776245117
time for calcul the mask position with numpy : 0.0016164779663085938
nb_pixel_total : 2370
time to create 1 rle with old method : 0.004168510437011719
time for calcul the mask position with numpy : 0.0016245841979980469
nb_pixel_total : 3952
time to create 1 rle with old method : 0.009892463684082031
time for calcul the mask position with numpy : 0.0018413066864013672
nb_pixel_total : 13002
time to create 1 rle with old method : 0.019840240478515625
time for calcul the mask position with numpy : 0.0015347003936767578
nb_pixel_total : 3174
time to create 1 rle with old method : 0.0040378570556640625
time for calcul the mask position with numpy : 0.0015211105346679688
nb_pixel_total : 3554
time to create 1 rle with old method : 0.004433631896972656
time for calcul the mask position with numpy : 0.0015363693237304688
nb_pixel_total : 16341
time to create 1 rle with old method : 0.01945805549621582
time for calcul the mask position with numpy : 0.0015521049499511719
nb_pixel_total : 2080
time to create 1 rle with old method : 0.0024721622467041016
time for calcul the mask position with numpy : 0.0015273094177246094
nb_pixel_total : 1692
time to create 1 rle with old method : 0.0021543502807617188
time for calcul the mask position with numpy : 0.001455545425415039
nb_pixel_total : 1190
time to create 1 rle with old method : 0.0015344619750976562
time for calcul the mask position with numpy : 0.00145721435546875
nb_pixel_total : 4279
time to create 1 rle with old method : 0.005530357360839844
time for calcul the mask position with numpy : 0.001537322998046875
nb_pixel_total : 7625
time to create 1 rle with old method : 0.009217262268066406
time for calcul the mask position with numpy : 0.0016968250274658203
nb_pixel_total : 27649
time to create 1 rle with old method : 0.033025503158569336
time for calcul the mask position with numpy : 0.0014827251434326172
nb_pixel_total : 933
time to create 1 rle with old method : 0.0013661384582519531
time for calcul the mask position with numpy : 0.0014421939849853516
nb_pixel_total : 4172
time to create 1 rle with old method : 0.005329132080078125
time for calcul the mask position with numpy : 0.001444101333618164
nb_pixel_total : 1507
time to create 1 rle with old method : 0.0018887519836425781
time for calcul the mask position with numpy : 0.0015316009521484375
nb_pixel_total : 7696
time to create 1 rle with old method : 0.010276079177856445
time for calcul the mask position with numpy : 0.0017116069793701172
nb_pixel_total : 38943
time to create 1 rle with old method : 0.04671931266784668
time for calcul the mask position with numpy : 0.001547098159790039
nb_pixel_total : 1634
time to create 1 rle with old method : 0.0021619796752929688
time for calcul the mask position with numpy : 0.00147247314453125
nb_pixel_total : 1597
time to create 1 rle with old method : 0.002103567123413086
time for calcul the mask position with numpy : 0.001468658447265625
nb_pixel_total : 2797
time to create 1 rle with old method : 0.0035157203674316406
time for calcul the mask position with numpy : 0.0015041828155517578
nb_pixel_total : 2447
time to create 1 rle with old method : 0.00305938720703125
time for calcul the mask position with numpy : 0.0015478134155273438
nb_pixel_total : 11971
time to create 1 rle with old method : 0.014048099517822266
time for calcul the mask position with numpy : 0.0015208721160888672
nb_pixel_total : 2726
time to create 1 rle with old method : 0.003311634063720703
time for calcul the mask position with numpy : 0.0015480518341064453
nb_pixel_total : 14659
time to create 1 rle with old method : 0.017230987548828125
time for calcul the mask position with numpy : 0.0017125606536865234
nb_pixel_total : 13073
time to create 1 rle with old method : 0.022606849670410156
time for calcul the mask position with numpy : 0.0016291141510009766
nb_pixel_total : 3344
time to create 1 rle with old method : 0.005769491195678711
time for calcul the mask position with numpy : 0.0015933513641357422
nb_pixel_total : 344
time to create 1 rle with old method : 0.0006709098815917969
time for calcul the mask position with numpy : 0.0016918182373046875
nb_pixel_total : 5531
time to create 1 rle with old method : 0.009562253952026367
time for calcul the mask position with numpy : 0.001819610595703125
nb_pixel_total : 10568
time to create 1 rle with old method : 0.017777204513549805
time for calcul the mask position with numpy : 0.0018126964569091797
nb_pixel_total : 18441
time to create 1 rle with old method : 0.02357959747314453
time for calcul the mask position with numpy : 0.0015177726745605469
nb_pixel_total : 4137
time to create 1 rle with old method : 0.005224466323852539
time for calcul the mask position with numpy : 0.0015132427215576172
nb_pixel_total : 1254
time to create 1 rle with old method : 0.0016112327575683594
time for calcul the mask position with numpy : 0.0015113353729248047
nb_pixel_total : 1030
time to create 1 rle with old method : 0.001332998275756836
time for calcul the mask position with numpy : 0.001440286636352539
nb_pixel_total : 888
time to create 1 rle with old method : 0.0012292861938476562
time for calcul the mask position with numpy : 0.001459360122680664
nb_pixel_total : 3854
time to create 1 rle with old method : 0.004938840866088867
time for calcul the mask position with numpy : 0.0015256404876708984
nb_pixel_total : 2418
time to create 1 rle with old method : 0.0031554698944091797
time for calcul the mask position with numpy : 0.0017163753509521484
nb_pixel_total : 38813
time to create 1 rle with old method : 0.04644942283630371
time for calcul the mask position with numpy : 0.0017468929290771484
nb_pixel_total : 8507
time to create 1 rle with old method : 0.01443624496459961
time for calcul the mask position with numpy : 0.0016765594482421875
nb_pixel_total : 875
time to create 1 rle with old method : 0.0016837120056152344
time for calcul the mask position with numpy : 0.0016825199127197266
nb_pixel_total : 2324
time to create 1 rle with old method : 0.00407719612121582
time for calcul the mask position with numpy : 0.0016627311706542969
nb_pixel_total : 594
time to create 1 rle with old method : 0.0011487007141113281
time for calcul the mask position with numpy : 0.0016665458679199219
nb_pixel_total : 2774
time to create 1 rle with old method : 0.0048809051513671875
time for calcul the mask position with numpy : 0.0016126632690429688
nb_pixel_total : 860
time to create 1 rle with old method : 0.0016531944274902344
time for calcul the mask position with numpy : 0.0015821456909179688
nb_pixel_total : 1143
time to create 1 rle with old method : 0.002090930938720703
time for calcul the mask position with numpy : 0.0016565322875976562
nb_pixel_total : 2408
time to create 1 rle with old method : 0.004429817199707031
time for calcul the mask position with numpy : 0.0016012191772460938
nb_pixel_total : 577
time to create 1 rle with old method : 0.0010614395141601562
time for calcul the mask position with numpy : 0.0015933513641357422
nb_pixel_total : 334
time to create 1 rle with old method : 0.0006654262542724609
time for calcul the mask position with numpy : 0.0015819072723388672
nb_pixel_total : 1668
time to create 1 rle with old method : 0.002957582473754883
time for calcul the mask position with numpy : 0.0016052722930908203
nb_pixel_total : 692
time to create 1 rle with old method : 0.001302957534790039
time for calcul the mask position with numpy : 0.0015730857849121094
nb_pixel_total : 584
time to create 1 rle with old method : 0.0010993480682373047
time for calcul the mask position with numpy : 0.0015959739685058594
nb_pixel_total : 1047
time to create 1 rle with old method : 0.0019512176513671875
time for calcul the mask position with numpy : 0.0016832351684570312
nb_pixel_total : 12970
time to create 1 rle with old method : 0.015669584274291992
time for calcul the mask position with numpy : 0.0015246868133544922
nb_pixel_total : 3147
time to create 1 rle with old method : 0.0041120052337646484
time for calcul the mask position with numpy : 0.0015330314636230469
nb_pixel_total : 1070
time to create 1 rle with old method : 0.0014104843139648438
time for calcul the mask position with numpy : 0.0015079975128173828
nb_pixel_total : 8602
time to create 1 rle with old method : 0.01060938835144043
time for calcul the mask position with numpy : 0.001628875732421875
nb_pixel_total : 16674
time to create 1 rle with old method : 0.020156383514404297
time for calcul the mask position with numpy : 0.0015256404876708984
nb_pixel_total : 4442
time to create 1 rle with old method : 0.0056400299072265625
time for calcul the mask position with numpy : 0.0015091896057128906
nb_pixel_total : 1739
time to create 1 rle with old method : 0.0021126270294189453
time for calcul the mask position with numpy : 0.0014853477478027344
nb_pixel_total : 712
time to create 1 rle with old method : 0.0009462833404541016
time for calcul the mask position with numpy : 0.001447439193725586
nb_pixel_total : 267
time to create 1 rle with old method : 0.00034737586975097656
time for calcul the mask position with numpy : 0.0014786720275878906
nb_pixel_total : 1327
time to create 1 rle with old method : 0.0016880035400390625
time for calcul the mask position with numpy : 0.0015077590942382812
nb_pixel_total : 969
time to create 1 rle with old method : 0.0012023448944091797
time for calcul the mask position with numpy : 0.0014712810516357422
nb_pixel_total : 9076
time to create 1 rle with old method : 0.010695457458496094
time for calcul the mask position with numpy : 0.0015325546264648438
nb_pixel_total : 10160
time to create 1 rle with old method : 0.012058734893798828
time for calcul the mask position with numpy : 0.0015187263488769531
nb_pixel_total : 616
time to create 1 rle with old method : 0.0007898807525634766
time for calcul the mask position with numpy : 0.0014214515686035156
nb_pixel_total : 1026
time to create 1 rle with old method : 0.0013191699981689453
time for calcul the mask position with numpy : 0.0014810562133789062
nb_pixel_total : 7504
time to create 1 rle with old method : 0.009117603302001953
time for calcul the mask position with numpy : 0.0014815330505371094
nb_pixel_total : 899
time to create 1 rle with old method : 0.001184225082397461
time for calcul the mask position with numpy : 0.0014338493347167969
nb_pixel_total : 970
time to create 1 rle with old method : 0.0014824867248535156
time for calcul the mask position with numpy : 0.0014407634735107422
nb_pixel_total : 222
time to create 1 rle with old method : 0.0003414154052734375
time for calcul the mask position with numpy : 0.0014488697052001953
nb_pixel_total : 736
time to create 1 rle with old method : 0.0010671615600585938
time for calcul the mask position with numpy : 0.0014557838439941406
nb_pixel_total : 1501
time to create 1 rle with old method : 0.0019741058349609375
time for calcul the mask position with numpy : 0.0014350414276123047
nb_pixel_total : 299
time to create 1 rle with old method : 0.0004889965057373047
time for calcul the mask position with numpy : 0.001438140869140625
nb_pixel_total : 246
time to create 1 rle with old method : 0.00036144256591796875
time for calcul the mask position with numpy : 0.0014352798461914062
nb_pixel_total : 1436
time to create 1 rle with old method : 0.0019245147705078125
time for calcul the mask position with numpy : 0.0014410018920898438
nb_pixel_total : 595
time to create 1 rle with old method : 0.0008265972137451172
time for calcul the mask position with numpy : 0.0015337467193603516
nb_pixel_total : 11111
time to create 1 rle with old method : 0.013346433639526367
time for calcul the mask position with numpy : 0.0014955997467041016
nb_pixel_total : 1327
time to create 1 rle with old method : 0.0017809867858886719
time for calcul the mask position with numpy : 0.0014755725860595703
nb_pixel_total : 2202
time to create 1 rle with old method : 0.0030286312103271484
time for calcul the mask position with numpy : 0.0015099048614501953
nb_pixel_total : 888
time to create 1 rle with old method : 0.0011479854583740234
time for calcul the mask position with numpy : 0.0014834403991699219
nb_pixel_total : 1129
time to create 1 rle with old method : 0.0014393329620361328
time for calcul the mask position with numpy : 0.0015223026275634766
nb_pixel_total : 929
time to create 1 rle with old method : 0.0012142658233642578
time for calcul the mask position with numpy : 0.0014948844909667969
nb_pixel_total : 885
time to create 1 rle with old method : 0.0011792182922363281
time for calcul the mask position with numpy : 0.0014674663543701172
nb_pixel_total : 1614
time to create 1 rle with old method : 0.002109050750732422
time for calcul the mask position with numpy : 0.0014460086822509766
nb_pixel_total : 1205
time to create 1 rle with old method : 0.0015971660614013672
time for calcul the mask position with numpy : 0.0014603137969970703
nb_pixel_total : 2325
time to create 1 rle with old method : 0.002905130386352539
insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s)
insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s)
batch 1
Loaded 101 chid ids of type : 4677
Number RLEs to save : 9710
INSERT IGNORE INTO MTRPhoto.crop_segments (`crop_hashtag_id`, `x0`, `y0`, `length`) VALUES (%s, %s, %s , %s) 
 first line : ('3689745357', '103', '565', '4')
...
 last line : ('3689745457', '932', '504', '3')
INSERT IGNORE INTO MTRPhoto.crop_sum_segments (`crop_hashtag_id`, `sum_segments`) VALUES (%s, %s) 
 TO DO : save crop sub photo not yet done ! 

After datou_step_exec type output : <class 'dict'>
map_portfolio_photo : len 0 keys : dict_keys([])
Inside saveOutput : final : True verbose : True
 saveOutput not yet implemented for datou_step.type : sam we use saveGeneral 
[1189321094]
 map_info['map_portfolio_photo'] : {}
 final :  True mtd_id 4573 list_pids : [1189321094]
Looping around the photos to save general results
len do output : 1
/1189321094Didn't retrieve data
.Didn't retrieve data
.
before output type
Here is an output not treated by saveGeneral : <mtr.datou.datou_lib_object.DatouData object at 0x7f0883ac2580>
Here is an output not treated by saveGeneral : <mtr.datou.datou_lib_object.DatouData object at 0x7f0883ac24c0>
Managing all output in save final without adding information in the mtr_datou_result 
('4573', None, None, None, None, None, None, None, None)
('4573', None, '1189321094', None, None, None, None, None, None)


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 3
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('4573', None, '1189321094', 'None', None, None, None, None, None)]
time used for this insertion : 0.01285409927368164
 save_final 

save missing photos in datou_result :

time spend for datou_step_exec : 18.417400360107422
time spend to save output : 0.014383316040039062
total time spend for step 1 : 18.43178367614746

caffe_path_current : 
About to save ! 2
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : {'1189321094': [[<mtr.simple_image_editor.image_utils.CHI object at 0x7f0828593d90>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828593dc0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828593d60>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828593d30>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828593c40>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828593c70>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828593ca0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c9a0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c580>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c850>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c8e0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c790>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c7c0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c6a0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c550>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c760>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c640>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c730>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c520>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c670>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c610>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c4f0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c460>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c430>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c3a0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c370>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c2e0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c2b0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c220>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c1f0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c160>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c130>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c100>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c0d0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082858c040>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585fd0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585fa0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585f70>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585ee0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585eb0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585e20>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585df0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585d60>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585d30>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585d00>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585cd0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585c40>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585c10>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585be0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585bb0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585b20>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585af0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585ac0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585a90>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585a00>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285859d0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285859a0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585970>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285858e0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285858b0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585820>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285857f0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585760>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585730>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585700>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285856d0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585640>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585610>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285855e0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285855b0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585520>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285854f0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285854c0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585490>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585400>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285853d0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285853a0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585370>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285852e0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285852b0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585280>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585250>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285851c0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585190>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585160>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585130>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f08285850a0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585070>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f0828585040>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857efd0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857ef40>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857ef10>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857eee0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857eeb0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857ee20>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857edf0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857ed60>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857ed30>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857eca0>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857ec70>, <mtr.simple_image_editor.image_utils.CHI object at 0x7f082857ebe0>], 'temp/1740728182_3374456_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png']}
nb_objects detect : 101

############################### TEST frcnn  ################################
test frcnn
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4184
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4184
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4184
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : frcnn
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (917754606) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 917754606
download finish for photo 917754606
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.15489792823791504
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:frcnn
Fri Feb 28 08:36:41 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1740728201_3374456_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606}
map_photo_id_path_extension : {917754606: {'path': 'temp/1740728201_3374456_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou step Faster rcnn !
classes : ['background', 'plaque']
pht : 4370
caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'}
To loadFromThcl()
model_param file didn't exist 
model_name : detection_plaque_valcor_010622
model_type : caffe_faster_rcnn
list file need :
['caffemodel', 'test.prototxt']
file exist in s3 :
['caffemodel', 'test.prototxt']
file manque in s3 : 
[]
local folder  : /data/models_weight/detection_plaque_valcor_010622


/data/models_weight/detection_plaque_valcor_010622/caffemodel
size_local : 349723073 size in s3 : 349723073
create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:56
caffemodel already exist and didn't need to update


/data/models_weight/detection_plaque_valcor_010622/test.prototxt
size_local : 7163 size in s3 : 7163
create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:55
test.prototxt already exist and didn't need to update
 prototxt : /data/models_weight/detection_plaque_valcor_010622/test.prototxt
 caffemodel : /data/models_weight/detection_plaque_valcor_010622/caffemodel


Loaded network /data/models_weight/detection_plaque_valcor_010622/caffemodel
About to compute detect_faster_rcnn : len(args) : 1
Inside frcnn step exec : nb paths : 1
image_path : temp/1740728201_3374456_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg
image_size (600, 800, 3)
[[[  4   6   6]
  [  5   7   7]
  [  6   8   8]
  ...
  [207 215 214]
  [206 214 213]
  [206 214 213]]

 [[  4   6   6]
  [  5   7   7]
  [  6   8   8]
  ...
  [207 215 214]
  [206 214 213]
  [206 214 213]]

 [[  4   6   6]
  [  5   7   7]
  [  6   8   8]
  ...
  [207 215 214]
  [206 214 213]
  [206 214 213]]

 ...

 [[ 14  16  16]
  [ 13  15  15]
  [ 11  13  13]
  ...
  [198 206 205]
  [198 206 205]
  [198 206 205]]

 [[ 16  18  18]
  [ 14  16  16]
  [ 11  13  13]
  ...
  [206 214 213]
  [206 214 213]
  [206 214 213]]

 [[ 13  15  15]
  [ 12  14  14]
  [  9  11  11]
  ...
  [210 218 217]
  [210 218 217]
  [210 218 217]]]
Detection took 0.076s for 300 object proposals
 c : plaque
 list_crops.shape (72, 5)
 proba :  0.063856065
(374.12598, 293.9191, 430.813, 317.81058)
 proba :  0.05222375
(382.17487, 297.18372, 552.3663, 344.653)
 proba :  0.0122770965
(345.3581, 272.42957, 468.8504, 320.726)
We are managing local photo_id
len de result frcnn : 1

After datou_step_exec type output : <class 'dict'>

time spend for datou_step_exec : 2.494955539703369
time spend to save output : 7.271766662597656e-05
total time spend for step 1 : 2.495028257369995

caffe_path_current : 
<module 'caffe' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/__init__.py'>
<module 'caffe.classifier' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/classifier.py'>
About to save ! 1
Inside saveOutput : final : True verbose : True
Inside saveFrcnn : final : True verbose : True
threshold to save the result : 0.1
output flattener :  [(0, 493029425, 4370, 374, 430, 293, 317, 0.063856065, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.05222375, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.0122770965, None)]
Warning : no hashtag_ids to insert in the database
final : True


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 1
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('4184', None, '917754606', '0', 0, '0', 493061979, '0', None)]
time used for this insertion : 0.01182246208190918
[917754606]
 map_info['map_portfolio_photo'] : {}
 final :  True mtd_id 4184 list_pids : [917754606]
Looping around the photos to save general results
len do output : 1
/0
before output type
Managing all output in save final without adding information in the mtr_datou_result 
('4184', None, None, None, None, None, None, None, None)
('4184', None, '917754606', None, None, None, None, None, None)


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 1
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('4184', None, '917754606', None, None, None, None, None, None)]
time used for this insertion : 0.010860443115234375
 save_final 

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : {0: [[(0, 493029425, 4370, 374, 430, 293, 317, 0.063856065, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.05222375, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.0122770965, None)], 'temp/1740728201_3374456_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg']}

############################### TEST thcl  ################################
TEST THCL
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=2

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=2
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 2
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=2
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
Here we check the consistency of inputs/outputs number between the given ones and the db !
eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering !
WARNING : step 1 thcl is not linked in the step_by_step architecture !
WARNING : step 2 argmax is not linked in the step_by_step architecture !
Number of inputs / outputs for each step checked !
Here we check the consistency of outputs/inputs types during steps connections
eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput !
DataTypes for each output/input checked !
no param json to modify
List Step Type Loaded in datou : thcl, argmax
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (916235064) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 916235064
download finish for photo 916235064
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.35868287086486816
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  2
step1:thcl
Fri Feb 28 08:36:44 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1740728204_3374456_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064}
map_photo_id_path_extension : {916235064: {'path': 'temp/1740728204_3374456_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou step Thcl !
multi_thcl or not :False
multi_thcl_cond or not :False
dic_thcl :  {'355': 1}
we are using the classfication for only one thcl
355


In convert_file_to_np l 337 : 1
 l343 1
l357 after caffe.io.load_image
dimension du image : (3, (66, 66, 3))
dimension displayed !
time to import caffe and check if the image exist :  0.006264209747314453
time to convert the images to numpy array :  0.004488706588745117
total time to convert the images to numpy array :  0.011229276657104492
list photo_ids error:
[]
list photo_ids correct :
[916235064]
number of photos to traite : 1
try to delete the photos incorrect in DB

tagging for thcl : 355
To do loadFromThcl(), then load ParamDescType : thcl355
 get_desc_type_from_thcl : type of cat <class 'mtr.database_queries.classification_admin_tools.ClassificationMetaTools'>

            SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists,
            photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at,
            type_classification
            FROM MTRDatou.classification_theme  WHERE  `id` IN (355) 
 thcls : [{'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3390
 SELECT * FROM MTRDatou.photo_desc_type_params  WHERE id in (3390)
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27))
To loadFromThcl() : net_3390
begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 2500
l 3632 free memory gpu now : 1207
wait 20 seconds
l 3637 free memory gpu now : 1207  max_wait_temp :  1 max_wait :  0
 SELECT * FROM MTRDatou.photo_desc_type_params  WHERE id in (3390)
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27))
 param : <mtr.database_queries.descriptor_queries.ParamDescriptorType object at 0x7f0615e92e80>, param.caffemodel : car_360_1027
None
 mean_file_type : 
 mean_file_path : 
 prototxt_file_path : 
 model : car_360_1027
 Inside get_net  
 Inside get_net before cache_data_model 
model_param file didn't exist 
 Inside get_net before  CDM.load_model_par_type  
model_name : car_360_1027
model_type : caffe
list file need :
['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt']
file exist in s3 :
['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt']
file manque in s3 : 
[]
local folder  : /data/models_weight/car_360_1027


/data/models_weight/car_360_1027/caffemodel
size_local : 542944640 size in s3 : 542944640
create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43
caffemodel already exist and didn't need to update


/data/models_weight/car_360_1027/deploy_conv_normal.prototxt
size_local : 4626 size in s3 : 4626
create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42
deploy_conv_normal.prototxt already exist and didn't need to update


/data/models_weight/car_360_1027/deploy_fc.prototxt
size_local : 1132 size in s3 : 1132
create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43
deploy_fc.prototxt already exist and didn't need to update


/data/models_weight/car_360_1027/deploy.prototxt
size_local : 5654 size in s3 : 5654
create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42
deploy.prototxt already exist and didn't need to update


/data/models_weight/car_360_1027/mean.npy
size_local : 1572944 size in s3 : 1572944
create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:55
mean.npy already exist and didn't need to update


/data/models_weight/car_360_1027/synset_words.txt
size_local : 13687 size in s3 : 13687
create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43
synset_words.txt already exist and didn't need to update
 Inside get_net after CDM.load_model_par_type  
  After if not only_with_local_cache:
/home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/
Here before set mode gpu
Doing nothing but we could set mode gpu
after set mode gpu
 prototxt_filename : /data/models_weight/car_360_1027/deploy.prototxt
 caffemodel_filename : /data/models_weight/car_360_1027/caffemodel
now we set caffe to gpu mode


 before predict
begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 2500
l 3632 free memory gpu now : 1205
wait 20 seconds
l 3637 free memory gpu now : 1205  max_wait_temp :  1 max_wait :  0
dict_keys(['prob', 'pool5'])
time used to do the prepocess of the images : 0.01423788070678711
time used to do the prediction : 0.1639697551727295


save descriptor for thcl : 355
(1, 512, 7, 7)
Got the blobs of the net
 to insert :  
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
code_as_byte_string:b'0000000000'|
time to traite the descriptors :  0.04823708534240723
Testing : ['916235064']
 In select_photos_meta_from_ids: 
        SELECT photo_id, url, FROM_UNIXTIME(uploaded_at), latitude, longitude, text
        FROM MTRBack.photos
        WHERE photo_id IN (916235064) 
result : {916235064: {'photo_id': 916235064, 'url': 'https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/2017/10/14/6293d1bb790dc6902450e7c572b7d10b.jpg', 'latitude': 0.0, 'longitude': 0.0, 'text': None}}
list_photo_exists : [916235064]
storage_type for insertDescriptorsMulti : 1
To insert : 916235064
time to insert the descriptors : 3.1263396739959717






After datou_step_exec type output : <class 'dict'>
map_portfolio_photo : len 0 keys : dict_keys([])
Inside saveOutput : final : False verbose : True
time used to find the portfolios of the photos
select button_legend_list from MTRDatou.classification_theme where id = 355

 SAVE THCL, output : {'916235064': [[('916235064', 'c_elysee_1027_gao__port_506302', 0.0018813187, 332, '355'), ('916235064', 'mokka_1027_gao__port_506374', 0.0011636211, 332, '355'), ('916235064', 'captur_1027_gao__port_506399', 0.0008156801, 332, '355'), ('916235064', 'sorento_1027_gao__port_506192', 0.0011772753, 332, '355'), ('916235064', 'navara_1027_gao__port_506205', 0.0025850998, 332, '355'), ('916235064', 'xc90_1027_gao__port_506350', 0.004170023, 332, '355'), ('916235064', 'saxo_1027_gao__port_506052', 0.0034804754, 332, '355'), ('916235064', 'trafic_1027_gao__port_506295', 0.0073676943, 332, '355'), ('916235064', 'punto_evo_1027_gao__port_506066', 0.0021885845, 332, '355'), ('916235064', '5_1027_gao__port_506117', 0.0005798737, 332, '355'), ('916235064', '250_1027_gao__port_506065', 0.0045917146, 332, '355'), ('916235064', 'd_max_1027_gao__port_506125', 0.0031583884, 332, '355'), ('916235064', 'panamera_1027_gao__port_506387', 0.0022505932, 332, '355'), ('916235064', 'alhambra_1027_gao__port_506381', 0.005320743, 332, '355'), ('916235064', 'x6_1027_gao__port_506349', 0.0011001299, 332, '355'), ('916235064', 'vitara_1027_gao__port_506328', 0.0054023997, 332, '355'), ('916235064', 'fiesta_1027_gao__port_506377', 0.0039190436, 332, '355'), ('916235064', 'qashqai_1027_gao__port_506286', 0.0014786283, 332, '355'), ('916235064', '147_1027_gao__port_506124', 0.0019781133, 332, '355'), ('916235064', 'c5_1027_gao__port_506172', 0.001244177, 332, '355'), ('916235064', 'q5_1027_gao__port_506206', 0.0015049521, 332, '355'), ('916235064', 'giulia_1027_gao__port_506178', 0.0021693932, 332, '355'), ('916235064', 'karl_1027_gao__port_506371', 0.0027079487, 332, '355'), ('916235064', 'mehari_1027_gao__port_506076', 0.004704336, 332, '355'), ('916235064', '911_1027_gao__port_506114', 0.0019418717, 332, '355'), ('916235064', '508_1027_gao__port_506329', 0.00095854246, 332, '355'), ('916235064', 'idea_1027_gao__port_506122', 0.00077006634, 332, '355'), ('916235064', 'megane_1027_gao__port_506220', 0.0019467891, 332, '355'), ('916235064', 'ghibli_1027_gao__port_506174', 0.0013724826, 332, '355'), ('916235064', 'touareg_1027_gao__port_506224', 0.0016203483, 332, '355'), ('916235064', 'i10_1027_gao__port_506232', 0.0013925397, 332, '355'), ('916235064', 'jumper_1027_gao__port_506234', 0.0100441035, 332, '355'), ('916235064', 'classe_clk_1027_gao__port_506173', 0.0010792696, 332, '355'), ('916235064', 'kuga_1027_gao__port_506181', 0.0008447048, 332, '355'), ('916235064', 'ct_1027_gao__port_506323', 0.0012521922, 332, '355'), ('916235064', 'leon_1027_gao__port_506326', 0.0025843624, 332, '355'), ('916235064', 'ds5_1027_gao__port_506376', 0.0012428642, 332, '355'), ('916235064', 'cordoba_1027_gao__port_506048', 0.0028647745, 332, '355'), ('916235064', 'classe_cla_1027_gao__port_506400', 0.0012948054, 332, '355'), ('916235064', 'jumpy_1027_gao__port_506179', 0.010338093, 332, '355'), ('916235064', 'avensis_1027_gao__port_506311', 0.0018768475, 332, '355'), ('916235064', 'juke_1027_gao__port_506325', 0.0011342375, 332, '355'), ('916235064', '4008_1027_gao__port_506402', 0.0015757948, 332, '355'), ('916235064', '190_series_1027_gao__port_506051', 0.0039812108, 332, '355'), ('916235064', 'serie_3_1027_gao__port_506294', 0.0028744773, 332, '355'), ('916235064', 'q7_1027_gao__port_506318', 0.0023356443, 332, '355'), ('916235064', 'glc_1027_gao__port_506303', 0.0012106454, 332, '355'), ('916235064', 'grand_vitara_1027_gao__port_506175', 0.0011446278, 332, '355'), ('916235064', 's40_1027_gao__port_506099', 0.0022339555, 332, '355'), ('916235064', 'toledo_1027_gao__port_506061', 0.0017464592, 332, '355'), ('916235064', '5008_1027_gao__port_506337', 0.004699479, 332, '355'), ('916235064', 'continental_1027_gao__port_506250', 0.0021914463, 332, '355'), ('916235064', 'coupe_1027_gao__port_506082', 0.0022632438, 332, '355'), ('916235064', 'iq_1027_gao__port_506166', 0.0018174835, 332, '355'), ('916235064', '407_1027_gao__port_506133', 0.00090565387, 332, '355'), ('916235064', 'touran_1027_gao__port_506308', 0.0020403322, 332, '355'), ('916235064', '300c_1027_gao__port_506078', 0.0025335946, 332, '355'), ('916235064', 'classe_gl_1027_gao__port_506340', 0.0044890754, 332, '355'), ('916235064', 'vivaro_1027_gao__port_506310', 0.003425354, 332, '355'), ('916235064', 'sl_1027_gao__port_506100', 0.0031359561, 332, '355'), ('916235064', 'elise_1027_gao__port_506121', 0.0010256271, 332, '355'), ('916235064', '1007_1027_gao__port_506070', 0.0015356254, 332, '355'), ('916235064', 'i40_1027_gao__port_506218', 0.0005914737, 332, '355'), ('916235064', 'bipper_tepee_1027_gao__port_506227', 0.004028887, 332, '355'), ('916235064', 'focus_1027_gao__port_506272', 0.0011586393, 332, '355'), ('916235064', 'primera_1027_gao__port_506147', 0.0012158159, 332, '355'), ('916235064', 'r4_1027_gao__port_506160', 0.014967006, 332, '355'), ('916235064', 'a8_1027_gao__port_506265', 0.0011321257, 332, '355'), ('916235064', 'boxer_1027_gao__port_506202', 0.010545549, 332, '355'), ('916235064', 's5_1027_gao__port_506222', 0.0011984656, 332, '355'), ('916235064', 'r21_1027_gao__port_506093', 0.0041858866, 332, '355'), ('916235064', 'c3_1027_gao__port_506257', 0.0023630005, 332, '355'), ('916235064', 'santa_fe_1027_gao__port_506208', 0.0016322947, 332, '355'), ('916235064', 'm4_1027_gao__port_506344', 0.0015568442, 332, '355'), ('916235064', 'safrane_1027_gao__port_506077', 0.001395933, 332, '355'), ('916235064', 'classe_gle_1027_gao__port_506395', 0.002198126, 332, '355'), ('916235064', '0_1027_gao__port_506094', 0.008828875, 332, '355'), ('916235064', 'ix35_1027_gao__port_506219', 0.0014613569, 332, '355'), ('916235064', 'carens_1027_gao__port_506298', 0.0008824953, 332, '355'), ('916235064', 'classe_a_1027_gao__port_506339', 0.0024712444, 332, '355'), ('916235064', 'ix20_1027_gao__port_506343', 0.0010092824, 332, '355'), ('916235064', 'note_1027_gao__port_506365', 0.001596244, 332, '355'), ('916235064', 'a5_1027_gao__port_506200', 0.00153309, 332, '355'), ('916235064', 'sx4_1027_gao__port_506348', 0.0014915921, 332, '355'), ('916235064', 'sandero_1027_gao__port_506198', 0.0014586106, 332, '355'), ('916235064', '3008_1027_gao__port_506385', 0.005645152, 332, '355'), ('916235064', 'q50_1027_gao__port_506239', 0.0011165055, 332, '355'), ('916235064', 'latitude_1027_gao__port_506236', 0.0008019344, 332, '355'), ('916235064', 'v40_1027_gao__port_506391', 0.0017143062, 332, '355'), ('916235064', 'xsara_1027_gao__port_506087', 0.0009823416, 332, '355'), ('916235064', 'grand_c_max_1027_gao__port_506342', 0.001795925, 332, '355'), ('916235064', 'swift_1027_gao__port_506149', 0.0015021353, 332, '355'), ('916235064', 'serie_1_1027_gao__port_506184', 0.0015142411, 332, '355'), ('916235064', 'xc70_1027_gao__port_506393', 0.0036189388, 332, '355'), ('916235064', 'master_1027_gao__port_506203', 0.007957403, 332, '355'), ('916235064', 'clio_1027_gao__port_506280', 0.0029575175, 332, '355'), ('916235064', 'duster_1027_gao__port_506216', 0.0007442915, 332, '355'), ('916235064', 'traveller_1027_gao__port_506403', 0.004293711, 332, '355'), ('916235064', 'tipo_1027_gao__port_506355', 0.0010930271, 332, '355'), ('916235064', 'rav_4_1027_gao__port_506332', 0.0013602998, 332, '355'), ('916235064', 'coccinelle_1027_gao__port_506259', 0.003494811, 332, '355'), ('916235064', 'spacetourer_1027_gao__port_506401', 0.0030968725, 332, '355'), ('916235064', 'xe_1027_gao__port_506357', 0.001447246, 332, '355'), ('916235064', 'ds3_1027_gao__port_506324', 0.0013093579, 332, '355'), ('916235064', 'mx_5_1027_gao__port_506098', 0.0025888216, 332, '355'), ('916235064', 'land_cruiser_1027_gao__port_506315', 0.0095330775, 332, '355'), ('916235064', 'classe_b_1027_gao__port_506335', 0.0017213717, 332, '355'), ('916235064', '806_1027_gao__port_506088', 0.0025622211, 332, '355'), ('916235064', 'rx_8_1027_gao__port_506046', 0.0036220076, 332, '355'), ('916235064', 'spark_1027_gao__port_506185', 0.0010076183, 332, '355'), ('916235064', '6_1027_gao__port_506171', 0.0011182578, 332, '355'), ('916235064', 'bravo_1027_gao__port_506080', 0.0014650348, 332, '355'), ('916235064', 'nx_1027_gao__port_506345', 0.0013682867, 332, '355'), ('916235064', 'sharan_1027_gao__port_506347', 0.0050928835, 332, '355'), ('916235064', 'x_type_1027_gao__port_506067', 0.0007802838, 332, '355'), ('916235064', 'jimny_1027_gao__port_506233', 0.0046059927, 332, '355'), ('916235064', 'wrangler_1027_gao__port_506225', 0.0018000815, 332, '355'), ('916235064', 'c_crosser_1027_gao__port_506312', 0.0015927489, 332, '355'), ('916235064', 'v70_1027_gao__port_506278', 0.001967743, 332, '355'), ('916235064', 'classe_e_1027_gao__port_506300', 0.0017369294, 332, '355'), ('916235064', 'classe_v_1027_gao__port_506258', 0.012732046, 332, '355'), ('916235064', 'm3_1027_gao__port_506182', 0.0023375957, 332, '355'), ('916235064', 'abarth_500_1027_gao__port_506226', 0.0040433975, 332, '355'), ('916235064', 'serie_6_1027_gao__port_506262', 0.0011316251, 332, '355'), ('916235064', 'modus_1027_gao__port_506146', 0.001829425, 332, '355'), ('916235064', '3_1027_gao__port_506113', 0.0015082662, 332, '355'), ('916235064', '405_1027_gao__port_506108', 0.008055381, 332, '355'), ('916235064', 'allroad_1027_gao__port_506297', 0.0010596903, 332, '355'), ('916235064', 'auris_1027_gao__port_506322', 0.0011525871, 332, '355'), ('916235064', 'galaxy_1027_gao__port_506143', 0.002515013, 332, '355'), ('916235064', 'giulietta_1027_gao__port_506363', 0.0008651711, 332, '355'), ('916235064', '106_1027_gao__port_506073', 0.0082695475, 332, '355'), ('916235064', 'classe_m_1027_gao__port_506154', 0.0030018752, 332, '355'), ('916235064', 'espace_1027_gao__port_506313', 0.0010645236, 332, '355'), ('916235064', 'panda_1027_gao__port_506189', 0.009029626, 332, '355'), ('916235064', 'rcz_1027_gao__port_506197', 0.0011292588, 332, '355'), ('916235064', '4007_1027_gao__port_506162', 0.00067927624, 332, '355'), ('916235064', 'classe_cl_1027_gao__port_506249', 0.0010860296, 332, '355'), ('916235064', 'leaf_1027_gao__port_506139', 0.0018036737, 332, '355'), ('916235064', 'octavia_1027_gao__port_506237', 0.0018603783, 332, '355'), ('916235064', 'ds4_1027_gao__port_506336', 0.0024156198, 332, '355'), ('916235064', 'freelander_1027_gao__port_506084', 0.0023474179, 332, '355'), ('916235064', 'evasion_1027_gao__port_506109', 0.0031141902, 332, '355'), ('916235064', 'punto_1027_gao__port_506106', 0.001949338, 332, '355'), ('916235064', '2cv_1027_gao__port_506045', 0.007973501, 332, '355'), ('916235064', 'x4_1027_gao__port_506392', 0.0017949995, 332, '355'), ('916235064', 'antara_1027_gao__port_506247', 0.0012469859, 332, '355'), ('916235064', 'murano_1027_gao__port_506316', 0.0006089358, 332, '355'), ('916235064', 'alto_1027_gao__port_506201', 0.009229994, 332, '355'), ('916235064', 'meriva_1027_gao__port_506353', 0.0013764681, 332, '355'), ('916235064', 'orlando_1027_gao__port_506305', 0.0018462854, 332, '355'), ('916235064', 'new_beetle_1027_gao__port_506050', 0.0011637815, 332, '355'), ('916235064', '306_1027_gao__port_506145', 0.0035069725, 332, '355'), ('916235064', 'tiguan_1027_gao__port_506362', 0.0026826886, 332, '355'), ('916235064', 's_type_1027_gao__port_506101', 0.0011381432, 332, '355'), ('916235064', 'c1_1027_gao__port_506128', 0.0027511606, 332, '355'), ('916235064', 'vectra_1027_gao__port_506044', 0.0011991771, 332, '355'), ('916235064', 'outlander_1027_gao__port_506317', 0.0017120582, 332, '355'), ('916235064', '307_1027_gao__port_506074', 0.002001323, 332, '355'), ('916235064', 'a6_s6_1027_gao__port_506134', 0.0016571061, 332, '355'), ('916235064', 'nemo_combi_1027_gao__port_506196', 0.0022659784, 332, '355'), ('916235064', 'berlingo_1027_gao__port_506194', 0.0046620215, 332, '355'), ('916235064', 'partner_1027_gao__port_506285', 0.003940578, 332, '355'), ('916235064', 'cayenne_1027_gao__port_506177', 0.0037974669, 332, '355'), ('916235064', 'quattroporte_1027_gao__port_506240', 0.0024441665, 332, '355'), ('916235064', 'c_max_1027_gao__port_506282', 0.0013124619, 332, '355'), ('916235064', 'fabia_1027_gao__port_506396', 0.005300201, 332, '355'), ('916235064', 'cx_3_1027_gao__port_506281', 0.0014462487, 332, '355'), ('916235064', 'x_trail_1027_gao__port_506264', 0.0018315162, 332, '355'), ('916235064', 'scirocco_1027_gao__port_506276', 0.0047905073, 332, '355'), ('916235064', 'matiz_1027_gao__port_506144', 0.0017558157, 332, '355'), ('916235064', 'tigra_1027_gao__port_506069', 0.00085429114, 332, '355'), ('916235064', 'escort_1027_gao__port_506091', 0.0048403456, 332, '355'), ('916235064', 'c2_1027_gao__port_506081', 0.0014904271, 332, '355'), ('916235064', 'mini_1027_gao__port_506168', 0.0011922211, 332, '355'), ('916235064', 'i30_1027_gao__port_506291', 0.00063251285, 332, '355'), ('916235064', 'picanto_1027_gao__port_506238', 0.002974578, 332, '355'), ('916235064', 'mito_1027_gao__port_506072', 0.0015075767, 332, '355'), ('916235064', 'impreza_1027_gao__port_506085', 0.0020192477, 332, '355'), ('916235064', 'kangoo_1027_gao__port_506235', 0.006582714, 332, '355'), ('916235064', 'a4_1027_gao__port_506193', 0.0019874072, 332, '355'), ('916235064', 'cayman_1027_gao__port_506268', 0.0018142053, 332, '355'), ('916235064', 'sportage_1027_gao__port_506148', 0.0014276985, 332, '355'), ('916235064', 'up_1027_gao__port_506356', 0.0068632136, 332, '355'), ('916235064', 'optima_1027_gao__port_506386', 0.00089174486, 332, '355'), ('916235064', 'defender_1027_gao__port_506229', 0.006722689, 332, '355'), ('916235064', 'serie_2_1027_gao__port_506256', 0.0022277222, 332, '355'), ('916235064', 'edge_1027_gao__port_506187', 0.0008747659, 332, '355'), ('916235064', 'r19_1027_gao__port_506110', 0.0049425103, 332, '355'), ('916235064', 'jetta_1027_gao__port_506304', 0.0036194846, 332, '355'), ('916235064', 'eos_1027_gao__port_506115', 0.0038926757, 332, '355'), ('916235064', 'accord_1027_gao__port_506214', 0.0020129764, 332, '355'), ('916235064', 'yaris_1027_gao__port_506334', 0.0032333606, 332, '355'), ('916235064', 'classe_cls_1027_gao__port_506289', 0.00078521564, 332, '355'), ('916235064', 'polo_1027_gao__port_506361', 0.0043106396, 332, '355'), ('916235064', 'serie_4_1027_gao__port_506366', 0.0011477742, 332, '355'), ('916235064', 'mini_cabriolet_1027_gao__port_506204', 0.0008379333, 332, '355'), ('916235064', 'prius_1027_gao__port_506190', 0.0011490087, 332, '355'), ('916235064', 'lodgy_1027_gao__port_506188', 0.002017229, 332, '355'), ('916235064', 'serie_7_1027_gao__port_506307', 0.0012479068, 332, '355'), ('916235064', 'c15_1027_gao__port_506055', 0.017710097, 332, '355'), ('916235064', 'kadjar_1027_gao__port_506389', 0.0012503895, 332, '355'), ('916235064', 'insignia_1027_gao__port_506364', 0.0016434704, 332, '355'), ('916235064', '308_1027_gao__port_506279', 0.0021236201, 332, '355'), ('916235064', 'roomster_1027_gao__port_506241', 0.0018015532, 332, '355'), ('916235064', '80_1027_gao__port_506057', 0.0046137567, 332, '355'), ('916235064', '309_1027_gao__port_506063', 0.013525687, 332, '355'), ('916235064', 'tucson_1027_gao__port_506320', 0.0021235852, 332, '355'), ('916235064', 'x3_1027_gao__port_506212', 0.0008978721, 332, '355'), ('916235064', 'xf_1027_gao__port_506263', 0.00111673, 332, '355'), ('916235064', '2008_1027_gao__port_506394', 0.0026410872, 332, '355'), ('916235064', 'passat_1027_gao__port_506306', 0.0014976795, 332, '355'), ('916235064', 'compass_1027_gao__port_506260', 0.0032563396, 332, '355'), ('916235064', 'twingo_1027_gao__port_506309', 0.0064983033, 332, '355'), ('916235064', 'micra_1027_gao__port_506221', 0.0035854045, 332, '355'), ('916235064', 'golf_1027_gao__port_506155', 0.0031934197, 332, '355'), ('916235064', 'soul_1027_gao__port_506176', 0.0012890099, 332, '355'), ('916235064', 'rapid_1027_gao__port_506398', 0.002591564, 332, '355'), ('916235064', 'forester_1027_gao__port_506360', 0.0022762876, 332, '355'), ('916235064', 'slk_1027_gao__port_506210', 0.0015845752, 332, '355'), ('916235064', 'forfour_1027_gao__port_506341', 0.002179566, 332, '355'), ('916235064', 'serie_5_1027_gao__port_506209', 0.0013696945, 332, '355'), ('916235064', 'xj_1027_gao__port_506170', 0.0026011777, 332, '355'), ('916235064', 'pajero_1027_gao__port_506097', 0.0052107996, 332, '355'), ('916235064', 'agila_1027_gao__port_506119', 0.0048491545, 332, '355'), ('916235064', 'a6_1027_gao__port_506163', 0.0018912898, 332, '355'), ('916235064', 'fox_1027_gao__port_506092', 0.00084653427, 332, '355'), ('916235064', 'boxster_1027_gao__port_506267', 0.0015941223, 332, '355'), ('916235064', 'altea_1027_gao__port_506246', 0.0021391988, 332, '355'), ('916235064', 'samurai_1027_gao__port_506047', 0.0062521826, 332, '355'), ('916235064', 'trax_1027_gao__port_506296', 0.0019439025, 332, '355'), ('916235064', 'getz_1027_gao__port_506058', 0.0016387177, 332, '355'), ('916235064', 'cherokee_1027_gao__port_506269', 0.0029809745, 332, '355'), ('916235064', 'koleos_1027_gao__port_506378', 0.0015591459, 332, '355'), ('916235064', 'z_series_1027_gao__port_506123', 0.0016566158, 332, '355'), ('916235064', 'ecosport_1027_gao__port_506271', 0.0013230676, 332, '355'), ('916235064', 'space_star_1027_gao__port_506277', 0.0021140003, 332, '355'), ('916235064', 'rs3_sportback_1027_gao__port_506207', 0.0019115615, 332, '355'), ('916235064', 'civic_1027_gao__port_506141', 0.0026898813, 332, '355'), ('916235064', 'talisman_1027_gao__port_506390', 0.0007612863, 332, '355'), ('916235064', 'f_pace_1027_gao__port_506314', 0.0016165163, 332, '355'), ('916235064', 'classe_c_1027_gao__port_506299', 0.001794237, 332, '355'), ('916235064', 'tt_1027_gao__port_506075', 0.0013934682, 332, '355'), ('916235064', 'pathfinder_1027_gao__port_506183', 0.0016513864, 332, '355'), ('916235064', '156_1027_gao__port_506157', 0.0015444249, 332, '355'), ('916235064', 'cx_5_1027_gao__port_506228', 0.0014415003, 332, '355'), ('916235064', 'scenic_1027_gao__port_506255', 0.0016085054, 332, '355'), ('916235064', 'yeti_1027_gao__port_506358', 0.0020916346, 332, '355'), ('916235064', 'mustang_1027_gao__port_506053', 0.010051838, 332, '355'), ('916235064', 'stilo_1027_gao__port_506060', 0.0010834065, 332, '355'), ('916235064', 'ateca_1027_gao__port_506382', 0.0017011826, 332, '355'), ('916235064', 'fiorino_1027_gao__port_506217', 0.009197423, 332, '355'), ('916235064', 'classe_glk_1027_gao__port_506290', 0.0017017962, 332, '355'), ('916235064', 'fortwo_1027_gao__port_506230', 0.0016010924, 332, '355'), ('916235064', 'cruze_1027_gao__port_506186', 0.0010052045, 332, '355'), ('916235064', '107_1027_gao__port_506213', 0.0016274963, 332, '355'), ('916235064', 'aygo_1027_gao__port_506248', 0.003243066, 332, '355'), ('916235064', 'rx_1027_gao__port_506354', 0.0010633463, 332, '355'), ('916235064', '500_1027_gao__port_506245', 0.0016356644, 332, '355'), ('916235064', 'bora_1027_gao__port_506104', 0.003816394, 332, '355'), ('916235064', 'transit_1027_gao__port_506111', 0.004861731, 332, '355'), ('916235064', 'pt_cruiser_1027_gao__port_506054', 0.001916522, 332, '355'), ('916235064', 'patrol_1027_gao__port_506068', 0.004240796, 332, '355'), ('916235064', 'r8_1027_gao__port_506156', 0.0012722225, 332, '355'), ('916235064', 'xm_1027_gao__port_506102', 0.0022677765, 332, '355'), ('916235064', 's60_1027_gao__port_506191', 0.0031990407, 332, '355'), ('916235064', 'aveo_1027_gao__port_506158', 0.003839647, 332, '355'), ('916235064', 'captiva_1027_gao__port_506159', 0.0017187491, 332, '355'), ('916235064', 'ax_1027_gao__port_506153', 0.006896189, 332, '355'), ('916235064', 'rexton_1027_gao__port_506107', 0.001302191, 332, '355'), ('916235064', 'camaro_1027_gao__port_506056', 0.0024903785, 332, '355'), ('916235064', 'ypsilon_1027_gao__port_506131', 0.0019544207, 332, '355'), ('916235064', 'delta_1027_gao__port_506165', 0.0014004406, 332, '355'), ('916235064', 'c4_1027_gao__port_506370', 0.0013007397, 332, '355'), ('916235064', 'zx_1027_gao__port_506161', 0.0045935344, 332, '355'), ('916235064', 'verso_1027_gao__port_506242', 0.00077215466, 332, '355'), ('916235064', 'superb_1027_gao__port_506327', 0.0019942827, 332, '355'), ('916235064', 'r5_1027_gao__port_506253', 0.0095452685, 332, '355'), ('916235064', 'caddy_1027_gao__port_506330', 0.013826791, 332, '355'), ('916235064', 'x5_1027_gao__port_506243', 0.001120639, 332, '355'), ('916235064', 'f_type_1027_gao__port_506231', 0.0008299057, 332, '355'), ('916235064', 'fusion_1027_gao__port_506096', 0.0012668102, 332, '355'), ('916235064', 'dokker_1027_gao__port_506331', 0.0053571914, 332, '355'), ('916235064', '205_1027_gao__port_506062', 0.0066849794, 332, '355'), ('916235064', 'macan_1027_gao__port_506195', 0.0015572613, 332, '355'), ('916235064', 'tourneo_1027_gao__port_506369', 0.0064010937, 332, '355'), ('916235064', '108_1027_gao__port_506384', 0.0052635404, 332, '355'), ('916235064', '9_3_1027_gao__port_506071', 0.0008374979, 332, '355'), ('916235064', 'mondeo_1027_gao__port_506116', 0.0014396139, 332, '355'), ('916235064', 'cr_v_1027_gao__port_506164', 0.0016411915, 332, '355'), ('916235064', 'c30_1027_gao__port_506090', 0.001748548, 332, '355'), ('916235064', 'pulsar_1027_gao__port_506397', 0.001201858, 332, '355'), ('916235064', 'ibiza_1027_gao__port_506273', 0.0037228984, 332, '355'), ('916235064', 'a1_1027_gao__port_506338', 0.0012346038, 332, '355'), ('916235064', 'matrix_1027_gao__port_506140', 0.0007076805, 332, '355'), ('916235064', 'carnival_1027_gao__port_506136', 0.0022812057, 332, '355'), ('916235064', 'xantia_1027_gao__port_506086', 0.0021959809, 332, '355'), ('916235064', 'terrano_1027_gao__port_506083', 0.0020293917, 332, '355'), ('916235064', 'q3_1027_gao__port_506275', 0.00112639, 332, '355'), ('916235064', 'hr_v_1027_gao__port_506283', 0.0017804413, 332, '355'), ('916235064', 'expert_1027_gao__port_506142', 0.007368925, 332, '355'), ('916235064', 'multivan_1027_gao__port_506383', 0.006505333, 332, '355'), ('916235064', 'venga_1027_gao__port_506380', 0.0008005178, 332, '355'), ('916235064', 'scudo_1027_gao__port_506129', 0.005592049, 332, '355'), ('916235064', 'laguna_1027_gao__port_506368', 0.0007134983, 332, '355'), ('916235064', 'vel_satis_1027_gao__port_506130', 0.002726792, 332, '355'), ('916235064', 'b_max_1027_gao__port_506367', 0.0017246988, 332, '355'), ('916235064', 'ignis_1027_gao__port_506292', 0.0043555815, 332, '355'), ('916235064', '159_1027_gao__port_506064', 0.0010784082, 332, '355'), ('916235064', 'grande_punto_1027_gao__port_506138', 0.0023632802, 332, '355'), ('916235064', 'logan_1027_gao__port_506167', 0.004396927, 332, '355'), ('916235064', 's_max_1027_gao__port_506223', 0.0012526402, 332, '355'), ('916235064', 'caravelle_1027_gao__port_506351', 0.0030296568, 332, '355'), ('916235064', 'adam_1027_gao__port_506079', 0.0010537665, 332, '355'), ('916235064', '406_1027_gao__port_506132', 0.0013575568, 332, '355'), ('916235064', 'q30_1027_gao__port_506293', 0.00097156665, 332, '355'), ('916235064', 'almera_1027_gao__port_506089', 0.0010239116, 332, '355'), ('916235064', 'corsa_1027_gao__port_506095', 0.002520213, 332, '355'), ('916235064', 'corolla_1027_gao__port_506120', 0.0026821743, 332, '355'), ('916235064', 'xc60_1027_gao__port_506388', 0.0018986207, 332, '355'), ('916235064', 'viano_1027_gao__port_506211', 0.0026944461, 332, '355'), ('916235064', 'pro_cee_d_1027_gao__port_506274', 0.0008319556, 332, '355'), ('916235064', 'a3_1027_gao__port_506321', 0.0037383735, 332, '355'), ('916235064', 'v50_1027_gao__port_506150', 0.00079203234, 332, '355'), ('916235064', 'voyager_1027_gao__port_506169', 0.0030528551, 332, '355'), ('916235064', 'corvette_1027_gao__port_506049', 0.0037230726, 332, '355'), ('916235064', 'rio_1027_gao__port_506379', 0.0017740283, 332, '355'), ('916235064', 'jazz_1027_gao__port_506252', 0.0015303518, 332, '355'), ('916235064', '200_1027_gao__port_506112', 0.004087493, 332, '355'), ('916235064', 'tts_1027_gao__port_506199', 0.0011863002, 332, '355'), ('916235064', 'zafira_1027_gao__port_506287', 0.0026955588, 332, '355'), ('916235064', 'asx_1027_gao__port_506266', 0.0011407273, 332, '355'), ('916235064', '607_1027_gao__port_506118', 0.0012528588, 332, '355'), ('916235064', '207_1027_gao__port_506103', 0.0015149466, 332, '355'), ('916235064', 'classe_s_1027_gao__port_506301', 0.0031658288, 332, '355'), ('916235064', 'c6_1027_gao__port_506105', 0.0017347554, 332, '355'), ('916235064', 'express_1027_gao__port_506137', 0.01672317, 332, '355'), ('916235064', 'classe_gla_1027_gao__port_506352', 0.001825393, 332, '355'), ('916235064', 'v60_1027_gao__port_506333', 0.0021457612, 332, '355'), ('916235064', 'ka_1027_gao__port_506180', 0.0014151983, 332, '355'), ('916235064', 'range_rover_1027_gao__port_506254', 0.0020554024, 332, '355'), ('916235064', 'discovery_1027_gao__port_506375', 0.0022965805, 332, '355'), ('916235064', 'classe_r_1027_gao__port_506270', 0.001394312, 332, '355'), ('916235064', 'transporter_1027_gao__port_506319', 0.011969908, 332, '355'), ('916235064', 'cee_d_1027_gao__port_506288', 0.0010548134, 332, '355'), ('916235064', 'zoe_1027_gao__port_506244', 0.0020712723, 332, '355'), ('916235064', 'i20_1027_gao__port_506284', 0.0017868447, 332, '355'), ('916235064', 'gtv_1027_gao__port_506059', 0.0057234284, 332, '355'), ('916235064', 's4_avant_1027_gao__port_506261', 0.002766614, 332, '355'), ('916235064', 'x1_1027_gao__port_506372', 0.0017146755, 332, '355'), ('916235064', 'autres_1027_gao__port_506127', 0.004825758, 332, '355'), ('916235064', '208_1027_gao__port_506359', 0.001868631, 332, '355'), ('916235064', 'c8_1027_gao__port_506135', 0.0012580998, 332, '355'), ('916235064', 'astra_1027_gao__port_506215', 0.0012625301, 332, '355'), ('916235064', '2_1027_gao__port_506151', 0.00092451187, 332, '355'), ('916235064', 'doblo_1027_gao__port_506251', 0.007465426, 332, '355'), ('916235064', '807_1027_gao__port_506152', 0.0007290913, 332, '355'), ('916235064', '206_1027_gao__port_506126', 0.0010385807, 332, '355'), ('916235064', 'a7_1027_gao__port_506373', 0.00069119426, 332, '355'), ('916235064', 'renegade_1027_gao__port_506346', 0.002141866, 332, '355')]]}


begin to insert list_values into class_photo_scores : 
length of list_valuse in save_photo_hashtag_id_thcl_score : 0
insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score)
time used for this insertion : 2.193450927734375e-05

save missing photos in datou_result :

time spend for datou_step_exec : 48.5142035484314
time spend to save output : 24.14395570755005
total time spend for step 1 : 72.65815925598145

step2:argmax
Fri Feb 28 08:37:57 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'dict'>

Here we display some param of map_info ! 
map_filenames : {'temp/1740728204_3374456_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064}
map_photo_id_path_extension : {916235064: {'path': 'temp/1740728204_3374456_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou_step Argmax !
calculate argmax for thcl : 355

After datou_step_exec type output : <class 'dict'>
map_portfolio_photo : len 0 keys : dict_keys([])
Inside saveOutput : final : True verbose : True
 photo_id : 916235064 output[photo_id] :  [('916235064', 'c15_1027_gao__port_506055', 0.017710097, 332, '355'), 'temp/1740728204_3374456_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg']


begin to insert list_values into photo_hahstag_ids : 
length of list_valuse in save_photo_hashtag_id_type : 1
insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s)
insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s)
 first line : ('916235064', '2049863950', '332')
...
 last line : ('916235064', '2049863950', '332')
time used for this insertion : 1.5692014694213867


begin to insert list_values into class_photo_scores : 
length of list_valuse in save_photo_hashtag_id_thcl_score : 1
insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score)
time used for this insertion : 2.4599409103393555
len list_finale : 1, len picture : 1


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 1
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('2', None, '916235064', 'c15_1027_gao__port_506055', None, None, '2049863950', '0.017710097', None)]
time used for this insertion : 1.139941930770874
saving photo_ids in datou_result
photo id not in port


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 0
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : []
time used for this insertion : 6.9141387939453125e-06

save missing photos in datou_result :

time spend for datou_step_exec : 0.0005691051483154297
time spend to save output : 5.169727563858032
total time spend for step 2 : 5.170296669006348

caffe_path_current : 
<module 'caffe' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/__init__.py'>
<module 'caffe.classifier' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/classifier.py'>
About to save ! 2
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 2
 output : {'916235064': [('916235064', 'c15_1027_gao__port_506055', 0.017710097, 332, '355'), 'temp/1740728204_3374456_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg']}

############################### TEST tfhub2  ################################
TEST TFHUB2
######################## test with use_multi_inputs=0 ########################
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4567

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4567
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4567
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4567
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
Here we check the consistency of inputs/outputs number between the given ones and the db !
eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering !
WARNING : step 12835 tfhub_classification2 is not linked in the step_by_step architecture !
WARNING : step 12836 argmax is not linked in the step_by_step architecture !
Number of inputs / outputs for each step checked !
Here we check the consistency of outputs/inputs types during steps connections
eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput !
DataTypes for each output/input checked !
no param json to modify
List Step Type Loaded in datou : tfhub_classification2, argmax
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (1171252784,1171252764,1171252487) 
 Found this number of photos: 3
##### Call download_photos : nb_thread : 5
begin to download photo : 1171252487
begin to download photo : 1171252764
begin to download photo : 1171252784
download finish for photo 1171252487
download finish for photo 1171252764
download finish for photo 1171252784
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3
##### After load_data_input 
time to download the photos : 0.2064807415008545
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  2
step1:tfhub_classification2
Fri Feb 28 08:38:02 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1740728282_3374456_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg': 1171252487, 'temp/1740728282_3374456_1171252764_29d5179a892cc50aadc9d67245534b59.jpg': 1171252764, 'temp/1740728282_3374456_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg': 1171252784}
map_photo_id_path_extension : {1171252487: {'path': 'temp/1740728282_3374456_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg', 'extension': 'jpg'}, 1171252764: {'path': 'temp/1740728282_3374456_1171252764_29d5179a892cc50aadc9d67245534b59.jpg', 'extension': 'jpg'}, 1171252784: {'path': 'temp/1740728282_3374456_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou_step TFHub with tf2 !
multi_thcl or not :False
multi_thcl_cond or not :False
dic_thcl :  {'3609': 1}
we are using the classfication for only one thcl
3609


begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 1207
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 1207
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 1428
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 235
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 1428
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 990
wait 20 seconds
2025-02-28 08:40:11.108947: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2025-02-28 08:40:11.109555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 08:40:11.109643: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 08:40:11.109692: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 08:40:11.112005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 08:40:11.112123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 08:40:11.115104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 08:40:11.116458: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 08:40:11.122030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 08:40:11.123319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 08:40:11.123783: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2025-02-28 08:40:11.155128: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz
2025-02-28 08:40:11.156839: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f0504000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2025-02-28 08:40:11.156884: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2025-02-28 08:40:11.159935: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xa0ef1720 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-02-28 08:40:11.159967: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5
2025-02-28 08:40:11.160873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2025-02-28 08:40:11.161003: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 08:40:11.161038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2025-02-28 08:40:11.161144: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2025-02-28 08:40:11.161191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2025-02-28 08:40:11.161249: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2025-02-28 08:40:11.161313: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2025-02-28 08:40:11.161378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 08:40:11.162478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2025-02-28 08:40:11.162560: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2025-02-28 08:40:11.162607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-02-28 08:40:11.162623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2025-02-28 08:40:11.162636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2025-02-28 08:40:11.163794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5)
l 3637 free memory gpu now : 990  max_wait_temp :  6 max_wait :  5
1 Physical GPUs, 1 Logical GPUs

tagging for thcl : 3609
To do loadFromThcl(), then load ParamDescType : thcl3609
 get_desc_type_from_thcl : type of cat <class 'mtr.database_queries.classification_admin_tools.ClassificationMetaTools'>

            SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists,
            photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at,
            type_classification
            FROM MTRDatou.classification_theme  WHERE  `id` IN (3609) 
 thcls : [{'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'}]
 thcl {'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'}
 Update svm_hashtag_type_desc : 5832
 SELECT * FROM MTRDatou.photo_desc_type_params  WHERE id in (5832)
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5832, 'tfhub_19_06_2023', 1280, 1280, 'tfhub_19_06_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 6, 19, 12, 55, 22), datetime.datetime(2023, 6, 19, 12, 55, 22))
model_name : tfhub_19_06_2023
model_param file didn't exist 
model_name : tfhub_19_06_2023
model_type : tf_classification2
list file need :
['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5']
file exist in s3 :
['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5']
file manque in s3 : 
[]
2025-02-28 08:40:17.991906: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 08:40:17.992498: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 08:40:17.993038: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 08:40:17.993542: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 08:40:17.994068: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.98G (2129957120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 08:40:17.994562: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.79G (1916961536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 08:40:17.995107: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.61G (1725265408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 08:40:17.995666: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.45G (1552738816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-02-28 08:40:17.996207: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.30G (1397465088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
local folder  : /data/models_weight/tfhub_19_06_2023


/data/models_weight/tfhub_19_06_2023/Confusion_Matrix.png
size_local : 57753 size in s3 : 57753
create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15
Confusion_Matrix.png already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/Precision_Recall_jrm.jpg
size_local : 79724 size in s3 : 79724
create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20
Precision_Recall_jrm.jpg already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/Precision_Recall_pcm.jpg
size_local : 83556 size in s3 : 83556
create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15
Precision_Recall_pcm.jpg already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/Precision_Recall_pcnc.jpg
size_local : 74107 size in s3 : 74107
create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20
Precision_Recall_pcnc.jpg already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/Precision_Recall_pehd.jpg
size_local : 72705 size in s3 : 72705
create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:20
Precision_Recall_pehd.jpg already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/Precision_Recall_tapis_vide.jpg
size_local : 70874 size in s3 : 70874
create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:15
Precision_Recall_tapis_vide.jpg already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/Result_Summary.txt
size_local : 642 size in s3 : 642
create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22
Result_Summary.txt already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/checkpoint
size_local : 99 size in s3 : 99
create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22
checkpoint already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00000-of-00002
size_local : 216488 size in s3 : 216488
create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22
model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00001-of-00002
size_local : 32279708 size in s3 : 32279708
create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:21
model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.index
size_local : 43546 size in s3 : 43546
create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:22
model_checkpoint.ckpt.index already exist and didn't need to update


/data/models_weight/tfhub_19_06_2023/model_weights.h5
size_local : 16499144 size in s3 : 16499144
create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:15
model_weights.h5 already exist and didn't need to update
ERROR in datou_step_exec, will save and exit !
assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec
    output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec
    return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2
    this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs)
  File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__
    self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module,
  File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 77, in create_tfhub_model
    hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module"),
  File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__
    self._func = load_module(handle, tags, self._load_options)
  File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module
    return module_v2.load(handle, tags=tags, options=set_load_options)
  File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load
    obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load
    return load_internal(export_dir, tags)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal
    loader = loader_cls(object_graph_proto,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__
    self._load_all()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all
    self._load_nodes()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes
    node, setter = self._recreate(proto, node_id)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate
    return factory[kind]()
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in <lambda>
    "variable": lambda: self._recreate_variable(proto.variable),
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable
    return variables.Variable(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__
    return cls._variable_v2_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call
    return previous_getter(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter
    return captured_getter(captured_previous, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator
    return resource_variable_ops.UninitializedVariable(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__
    handle = _variable_handle_from_shape_and_dtype(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype
    gen_logging_ops._assert(  # pylint: disable=protected-access
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from

[1171252487, 1171252764, 1171252784]
 map_info['map_portfolio_photo'] : {}
 final :  True mtd_id 4567 list_pids : [1171252487, 1171252764, 1171252784]


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 3
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('4567', None, '1171252487', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 156 in test_tfhub2>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 819 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2346 in datou_exec>, 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252764', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 156 in test_tfhub2>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 819 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2346 in datou_exec>, 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252784', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 156 in test_tfhub2>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 819 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2346 in datou_exec>, 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)]
time used for this insertion : 0.03714799880981445
 save_final 
ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse

time spend for datou_step_exec : 135.56073021888733
time spend to save output : 0.04234027862548828
total time spend for step 0 : 135.60307049751282
need to delete datou_research and reload, so keep current state 1
caffe_path_current : 
<module 'caffe' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/__init__.py'>
<module 'caffe.classifier' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/classifier.py'>
About to save ! 2
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 2
 output : None
probably due to empty image bug
ERROR
expected : {'1171252784': [(1171252784, 'jrm', 0.9677492, 4674, '3609'), 'temp/1687511175_1882837_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg'], '1171252764': [(1171252764, 'jrm', 0.9853587, 4674, '3609'), 'temp/1687511175_1882837_1171252764_29d5179a892cc50aadc9d67245534b59.jpg'], '1171252487': [(1171252487, 'jrm', 0.9262757, 4674, '3609'), 'temp/1687511175_1882837_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg']}
got : None
######################## test with use_multi_inputs=1 ########################
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4621

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=4621
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 4621
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=4621
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
Here we check the consistency of inputs/outputs number between the given ones and the db !
eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering !
WARNING : step 12927 tfhub_classification2 is not linked in the step_by_step architecture !
WARNING : step 12928 argmax is not linked in the step_by_step architecture !
Number of inputs / outputs for each step checked !
Here we check the consistency of outputs/inputs types during steps connections
eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput !
DataTypes for each output/input checked !
no param json to modify
List Step Type Loaded in datou : tfhub_classification2, argmax
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (1171291875,1171275372,1171275314) 
 Found this number of photos: 3
##### Call download_photos : nb_thread : 5
begin to download photo : 1171275314
begin to download photo : 1171275372
begin to download photo : 1171291875
download finish for photo 1171275314
download finish for photo 1171291875
download finish for photo 1171275372
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3
##### After load_data_input 
time to download the photos : 0.7415266036987305
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  2
step1:tfhub_classification2
Fri Feb 28 08:40:19 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1740728418_3374456_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg': 1171275314, 'temp/1740728418_3374456_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg': 1171291875, 'temp/1740728418_3374456_1171275372_76d81364ff7df843bff095f45c07ba35.jpg': 1171275372}
map_photo_id_path_extension : {1171275314: {'path': 'temp/1740728418_3374456_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg', 'extension': 'jpg'}, 1171291875: {'path': 'temp/1740728418_3374456_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg', 'extension': 'jpg'}, 1171275372: {'path': 'temp/1740728418_3374456_1171275372_76d81364ff7df843bff095f45c07ba35.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou_step TFHub with tf2 !
multi_thcl or not :False
multi_thcl_cond or not :False
dic_thcl :  {'3655': 1}
we are using the classfication for only one thcl
3655


begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 7
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 7
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 7
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 228
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 228
wait 20 seconds
inside check gpu memory
havn't enough memory gpu , need / 3096
l 3632 free memory gpu now : 228
wait 20 seconds
l 3637 free memory gpu now : 228  max_wait_temp :  6 max_wait :  5
1 Physical GPUs, 1 Logical GPUs

tagging for thcl : 3655
To do loadFromThcl(), then load ParamDescType : thcl3655
 get_desc_type_from_thcl : type of cat <class 'mtr.database_queries.classification_admin_tools.ClassificationMetaTools'>

            SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists,
            photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at,
            type_classification
            FROM MTRDatou.classification_theme  WHERE  `id` IN (3655) 
 thcls : [{'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'}]
 thcl {'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'}
 Update svm_hashtag_type_desc : 5862
 SELECT * FROM MTRDatou.photo_desc_type_params  WHERE id in (5862)
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5862, 'tfhub_18_7_2023', 1280, 1280, 'tfhub_18_7_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 7, 18, 22, 46, 29), datetime.datetime(2023, 7, 18, 22, 46, 29))
model_name : tfhub_18_7_2023
model_param file didn't exist 
model_name : tfhub_18_7_2023
model_type : tf_classification2
list file need :
['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5']
file exist in s3 :
['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5']
file manque in s3 : 
[]
2025-02-28 08:42:37.516098: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2025-02-28 08:42:37.782173: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2025-02-28 08:42:37.785908: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
local folder  : /data/models_weight/tfhub_18_7_2023


/data/models_weight/tfhub_18_7_2023/Confusion_Matrix.png
size_local : 54360 size in s3 : 54360
create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:28
Confusion_Matrix.png already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/Precision_Recall_jrm.jpg
size_local : 72583 size in s3 : 72583
create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23
Precision_Recall_jrm.jpg already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/Precision_Recall_pcm.jpg
size_local : 81681 size in s3 : 81681
create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:17
Precision_Recall_pcm.jpg already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/Precision_Recall_pcnc.jpg
size_local : 79510 size in s3 : 79510
create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23
Precision_Recall_pcnc.jpg already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/Precision_Recall_pehd.jpg
size_local : 59936 size in s3 : 59936
create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23
Precision_Recall_pehd.jpg already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/Precision_Recall_tapis_vide.jpg
size_local : 78974 size in s3 : 78974
create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17
Precision_Recall_tapis_vide.jpg already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/Result_Summary.txt
size_local : 642 size in s3 : 642
create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23
Result_Summary.txt already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/checkpoint
size_local : 99 size in s3 : 99
create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23
checkpoint already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00000-of-00002
size_local : 216529 size in s3 : 216529
create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17
model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00001-of-00002
size_local : 32279748 size in s3 : 32279748
create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19
model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.index
size_local : 43546 size in s3 : 43546
create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19
model_checkpoint.ckpt.index already exist and didn't need to update


/data/models_weight/tfhub_18_7_2023/model_weights.h5
size_local : 16500868 size in s3 : 16500868
create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:18
model_weights.h5 already exist and didn't need to update
desc size : 1280
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_2 (InputLayer)            [(None, 224, 224, 3) 0                                            
__________________________________________________________________________________________________
input_3 (InputLayer)            [(None, 1)]          0                                            
__________________________________________________________________________________________________
module (KerasLayer)             (None, 1280)         4049564     input_2[0][0]                    
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 1281)         0           input_3[0][0]                    
                                                                 module[0][0]                     
__________________________________________________________________________________________________
tfhub_18_7_2023dense (Dense)    (None, 5)            6410        concatenate[0][0]                
==================================================================================================
Total params: 4,055,974
Trainable params: 0
Non-trainable params: 4,055,974
__________________________________________________________________________________________________
Loading Weights...
time used to create the model : 9.045474767684937
time used to load_weights : 0.12876296043395996
found 3 data
found 0 labels

 begin to do the prediction :
ERROR in datou_step_exec, will save and exit !
 Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node model_1/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/stem_conv2d/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_59661]

Function call stack:
predict_function

  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec
    output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec
    return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose)
  File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2
    classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio,
  File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths
    Y_pred, F_pred = self.model.predict(valid_generator, validation_steps)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict
    tmp_batch_outputs = predict_function(iterator)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call
    return self._call_flat(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call
    outputs = execute.execute(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,

[1171275314, 1171291875, 1171275372]
 map_info['map_portfolio_photo'] : {}
 final :  True mtd_id 4621 list_pids : [1171275314, 1171291875, 1171275372]


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 3
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('4621', None, '1171275314', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 202 in test_tfhub2>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 819 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2346 in datou_exec>, ' Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.\\n\\t [[{{node model_1/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/stem_conv2d/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_59661]\\n\\nFunction call stack:\\npredict_function\\n']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171291875', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 202 in test_tfhub2>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 819 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2346 in datou_exec>, ' Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.\\n\\t [[{{node model_1/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/stem_conv2d/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_59661]\\n\\nFunction call stack:\\npredict_function\\n']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171275372', "[<FrameSummary file /home/admin/workarea/git/Velours/python/tests/python_tests.py, line 245 in <module>>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 3680 in test>, <FrameSummary file /home/admin/workarea/git/Velours/python/tests/datou_test.py, line 202 in test_tfhub2>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 819 in batchDatouExec>, <FrameSummary file /home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py, line 2346 in datou_exec>, ' Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.\\n\\t [[{{node model_1/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/stem_conv2d/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_59661]\\n\\nFunction call stack:\\npredict_function\\n']", '-1', '-1.0', '501120777', '1.0', None)]
time used for this insertion : 0.5139904022216797
 save_final 
ERROR in last step tfhub_classification2,  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node model_1/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/stem_conv2d/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_59661]

Function call stack:
predict_function


time spend for datou_step_exec : 138.76502537727356
time spend to save output : 0.5198121070861816
total time spend for step 0 : 139.28483748435974
need to delete datou_research and reload, so keep current state 1
caffe_path_current : 
<module 'caffe' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/__init__.py'>
<module 'caffe.classifier' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/classifier.py'>
About to save ! 2
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 2
 output : None
probably due to empty image bug
ERROR
expected : {'1171291875': [(1171291875, 'tapis_vide', 0.97062814, 4723, '3655'), 'temp/1691745841_1143057_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg'], '1171275372': [(1171275372, 'tapis_vide', 0.9674145, 4723, '3655'), 'temp/1691745841_1143057_1171275372_76d81364ff7df843bff095f45c07ba35.jpg'], '1171275314': [(1171275314, 'tapis_vide', 0.96509415, 4723, '3655'), 'temp/1691745841_1143057_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg']}
got : None
ERROR tfhub2 FAILED

############################### TEST ordonner  ################################
To do loadFromThcl(), then load ParamDescType : thcl358
 get_desc_type_from_thcl : type of cat <class 'mtr.database_queries.classification_admin_tools.ClassificationMetaTools'>

            SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists,
            photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at,
            type_classification
            FROM MTRDatou.classification_theme  WHERE  `id` IN (358) 
 thcls : [{'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}]
 thcl {'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}
 Update svm_hashtag_type_desc : 3392
['FirstUploadExperveo_vignette__port_505674', 'CAR_EXTERIEUR_Roue__port_503398', 'FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486', 'FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465', 'CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198', 'CAR_EXTERIEUR_Face_avant_axe_droit__port_504451', 'CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235', 'FirstUploadExperveo_vin__port_505675', 'CAR_EXTERIEUR_cote_droite__port_504108', 'CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565', 'FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201', 'cartegrise_orientation__port_505064', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217', 'CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531', 'CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218', 'CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214', 'CAR_EXTERIEUR_Angle_avant_droit__port_504087', 'FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484', 'CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563', 'CAR_EXTERIEUR_Angle_arriere_droit__port_504160', 'CAR_EXTERIEUR_arriere__port_504184', 'CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562', 'INTERIEUR_Compteur_kilometrique__port_503644', 'CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494', 'CAR_EXTERIEUR_Angle_arriere_gauche__port_504170', 'CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226', 'CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202', 'CAR_EXTERIEUR_moteur__port_503704', 'FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487', 'CAR_INTERIEUR_siege_arriere_class_1__port_506551', 'CAR_EXTERIEUR_avant__port_504146', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215', 'CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225', 'CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564', 'FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'CAR_INTERIEUR_coffre__port_503412', 'FirstUploadExperveo_rouetranche__port_505677', 'UploadPhotoImmatBest_class_1__port_505051', 'CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532', 'CAR_EXTERIEUR_angle_avant_gauche__port_504098', 'CAR_EXTERIEUR_face_avant_axe_gauche__port_504236', 'CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540', 'CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233', 'CAR_EXTERIEUR_roue_de_secour__port_503763', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199', 'CAR_EXTERIEUR_cote_gauche__port_504017', 'CAR_INTERIEUR_avant_volant_class_1__port_506503', 'CAR_INTERIEUR_avant_volant_class_2__port_506504', 'CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234']
51
SELECT hashtag_id,hashtag FROM MTRBack.hashtags where hashtag in ('FirstUploadExperveo_vignette__port_505674','CAR_EXTERIEUR_Roue__port_503398','FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486','FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485','CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465','CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198','CAR_EXTERIEUR_Face_avant_axe_droit__port_504451','CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235','FirstUploadExperveo_vin__port_505675','CAR_EXTERIEUR_cote_droite__port_504108','CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565','FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483','CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201','cartegrise_orientation__port_505064','CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217','CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531','CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218','CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214','CAR_EXTERIEUR_Angle_avant_droit__port_504087','FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484','CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563','CAR_EXTERIEUR_Angle_arriere_droit__port_504160','CAR_EXTERIEUR_arriere__port_504184','CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562','INTERIEUR_Compteur_kilometrique__port_503644','CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494','CAR_EXTERIEUR_Angle_arriere_gauche__port_504170','CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226','CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202','CAR_EXTERIEUR_moteur__port_503704','FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487','CAR_INTERIEUR_siege_arriere_class_1__port_506551','CAR_EXTERIEUR_avant__port_504146','CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215','CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225','CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564','FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482','CAR_INTERIEUR_coffre__port_503412','FirstUploadExperveo_rouetranche__port_505677','UploadPhotoImmatBest_class_1__port_505051','CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532','CAR_EXTERIEUR_angle_avant_gauche__port_504098','CAR_EXTERIEUR_face_avant_axe_gauche__port_504236','CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540','CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233','CAR_EXTERIEUR_roue_de_secour__port_503763','CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199','CAR_EXTERIEUR_cote_gauche__port_504017','CAR_INTERIEUR_avant_volant_class_1__port_506503','CAR_INTERIEUR_avant_volant_class_2__port_506504','CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'); 
51
dict_keys(['cartegrise_orientation__port_505064', 'car_exterieur_angle_arriere_droit_axe_arriere__port_504217', 'car_exterieur_angle_arriere_droit_axe_droit__port_504215', 'car_exterieur_angle_arriere_droit__port_504160', 'car_exterieur_angle_arriere_gauche_axe_arriere__port_504201', 'car_exterieur_angle_arriere_gauche_axe_gauche__port_504199', 'car_exterieur_angle_arriere_gauche__port_504170', 'car_exterieur_angle_avant_droit_axe_arriere__port_504226', 'car_exterieur_angle_avant_droit_axe_droit__port_504225', 'car_exterieur_angle_avant_droit__port_504087', 'car_exterieur_angle_avant_gauche_axe_avant__port_504235', 'car_exterieur_angle_avant_gauche_axe_gauche__port_504234', 'car_exterieur_angle_avant_gauche__port_504098', 'car_exterieur_arriere__port_504184', 'car_exterieur_avant__port_504146', 'car_exterieur_cote_droite__port_504108', 'car_exterieur_cote_droit_axe_arriere__port_504214', 'car_exterieur_cote_droit_axe_avant__port_504465', 'car_exterieur_cote_gauche_axe_arriere__port_504198', 'car_exterieur_cote_gauche_axe_avant__port_504233', 'car_exterieur_cote_gauche__port_504017', 'car_exterieur_face_arriere_axe_droit__port_504218', 'car_exterieur_face_arriere_axe_gauche__port_504202', 'car_exterieur_face_avant_axe_droit__port_504451', 'car_exterieur_face_avant_axe_gauche__port_504236', 'car_exterieur_moteur__port_503704', 'car_exterieur_roue_de_secour__port_503763', 'car_exterieur_roue__port_503398', 'car_interieur_avant_volant_class_1__port_506503', 'car_interieur_avant_volant_class_2__port_506504', 'car_interieur_avant_volant_class_6_boutonrond__port_506562', 'car_interieur_avant_volant_class_6_class_2__port_506563', 'car_interieur_avant_volant_class_6_ecrangrosplan__port_506564', 'car_interieur_avant_volant_class_6_levierdevitesse__port_506565', 'car_interieur_avant_vue-arriere_class_1__port_506531', 'car_interieur_avant_vue-arriere_class_2__port_506532', 'car_interieur_avant_vue_droite_habitacle_class_1__port_506540', 'car_interieur_avant_vue_gauche_habitacle_class_1__port_506494', 'car_interieur_coffre__port_503412', 'car_interieur_siege_arriere_class_1__port_506551', 'firstuploadexperveo_carrosseriegrosplan_carrosserie__port_506483', 'firstuploadexperveo_carrosseriegrosplan_class_6__port_506487', 'firstuploadexperveo_carrosseriegrosplan_morceauderoue__port_506484', 'firstuploadexperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'firstuploadexperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'firstuploadexperveo_carrosseriegrosplan_vindanslamoquette__port_506486', 'firstuploadexperveo_rouetranche__port_505677', 'firstuploadexperveo_vignette__port_505674', 'firstuploadexperveo_vin__port_505675', 'interieur_compteur_kilometrique__port_503644', 'uploadphotoimmatbest_class_1__port_505051'])
select photo_hashtag_type from MTRDatou.classification_theme where id = 358
thcl :  358
photo_hashtag_type :  337

                    SELECT phi.hashtag_id , phi.photo_id FROM MTRBack.photo_hashtag_ids phi,
                    MTRUser.mtr_portfolio_photos mp
                    where phi.type = 337 and phi.photo_id = mp.mtr_photo_id and mp.mtr_portfolio_id =510365;
{510365: [(917973295, 1), (917973297, 1), (917973302, 1), (917973293, 7), (917973296, 11), (917973300, 11), (917973286, 13), (917973289, 13), (917973301, 24), (917973285, 29), (917973290, 29), (917973299, 29), (917973304, 35), (917973287, 36), (917973298, 36), (917973305, 36), (917973292, 37), (917973291, 41), (917973303, 41), (917973294, 42), (917973288, 46)]}

############################### TEST rotate  ################################
test rotate only
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=230

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=230
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 230
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=230
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
 DONE and to test : checkNoCycle !
We are managing only one step so we do not consider checkConsistencyNbInputNbOutput !
We are managing only one step so we do not consider checkConsistencyTypeOutputInput !
no param json to modify
List Step Type Loaded in datou : rotate
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (917849322) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 917849322
download finish for photo 917849322
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.2440938949584961
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  1
step1:rotate
Fri Feb 28 08:43:31 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322}
map_photo_id_path_extension : {917849322: {'path': 'temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou_step_rotate !
We are in a linear step without datou_depend !
rotate photos of 90,180,270 degres
batch 1
select photo_id, hashtag_id, `type`, x0, x1, y0, y1, score, id from MTRPhoto.crop_hashtag_ids  where photo_id in ( 917849322) and `type` in (0) 
Loaded 0 chid ids of type : 0
 SELECT crop_hashtag_id, points FROM MTRPhoto.crop_polygon_points WHERE crop_hashtag_id in ()
 map_chi : {}
 photo_id in download_rotate_and_save : 917849322 list_chi_loc : 0
Use all angle !
Rotation of photo 917849322 of 90 degree
temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 90
 remove_crop_border : False
version de PIL : 9.5.0
Needs to change image size !
[[ 6.123234e-17  1.000000e+00]
 [-1.000000e+00  6.123234e-17]]
90
[[ 6.123234e-17  1.000000e+00]
 [-1.000000e+00  6.123234e-17]]
 shrink_image : False
 image_rotate :  <PIL.Image.Image image mode=RGB size=2448x3264 at 0x7F03D42FA9A0>
 image_path : temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg
 path_name_rotate : temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg
 image_rotate.mode : RGB
Rotation of photo 917849322 of 180 degree
temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 180
 remove_crop_border : False
version de PIL : 9.5.0
Needs to change image size !
[[-1.0000000e+00  1.2246468e-16]
 [-1.2246468e-16 -1.0000000e+00]]
180
[[-1.0000000e+00  1.2246468e-16]
 [-1.2246468e-16 -1.0000000e+00]]
 shrink_image : False
 image_rotate :  <PIL.Image.Image image mode=RGB size=3264x2448 at 0x7F03D42FA9A0>
 image_path : temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg
 path_name_rotate : temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg
 image_rotate.mode : RGB
Rotation of photo 917849322 of 270 degree
temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 270
 remove_crop_border : False
version de PIL : 9.5.0
Needs to change image size !
[[-1.8369702e-16 -1.0000000e+00]
 [ 1.0000000e+00 -1.8369702e-16]]
270
[[-1.8369702e-16 -1.0000000e+00]
 [ 1.0000000e+00 -1.8369702e-16]]
 shrink_image : False
 image_rotate :  <PIL.Image.Image image mode=RGB size=2448x3264 at 0x7F03D42FA9A0>
 image_path : temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg
 path_name_rotate : temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg
 image_rotate.mode : RGB

About to upload 3 photos
upload in portfolio : 551782
init cache_photo without model_param  
we have 3 photo to upload
uploaded to storage server : ovh
folder_temporaire : temp/1740728612_3374456
we have uploaded 3 photos in the portfolio 551782

time of upload the photos Elapsed time : 2.395246744155884

 map_filename_photo_id : 3
 map_filename_photo_id : {'temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg': 1340097567, 'temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg': 1340097569, 'temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg': 1340097570}
Len new_chis : 3
Len list_new_chi_with_photo_id : 0 of type : 0
 list_new_chi_with_photo_id : []

After datou_step_exec type output : <class 'dict'>

time spend for datou_step_exec : 2.66510272026062
time spend to save output : 6.198883056640625e-05
total time spend for step 1 : 2.6651647090911865

caffe_path_current : 
<module 'caffe' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/__init__.py'>
<module 'caffe.classifier' from '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python/caffe/classifier.py'>
About to save ! 1
Inside saveOutput : final : True verbose : True
 saveOutput not yet implemented for datou_step.type : rotate we use saveGeneral 
[917849322]
 map_info['map_portfolio_photo'] : {}
 final :  True mtd_id 230 list_pids : [917849322]
Looping around the photos to save general results
len do output : 3
/1340097567Didn't retrieve data
.Didn't retrieve data
.Didn't retrieve data
.
/1340097569Didn't retrieve data
.Didn't retrieve data
.Didn't retrieve data
.
/1340097570Didn't retrieve data
.Didn't retrieve data
.Didn't retrieve data
.
before output type
Here is an output not treated by saveGeneral : <mtr.datou.datou_lib_object.DatouData object at 0x7f04444e6640>
Here is an output not treated by saveGeneral : <mtr.datou.datou_lib_object.DatouData object at 0x7f0444499610>
Here is an output not treated by saveGeneral : <mtr.datou.datou_lib_object.DatouData object at 0x7f0427d12820>
Managing all output in save final without adding information in the mtr_datou_result 
('230', None, None, None, None, None, None, None, None)
('230', None, '917849322', None, None, None, None, None, None)


begin to insert list_values into mtr_datou_result : 
length of list_values in save_final : 10
insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id
 list_values : [('230', None, '1340097567', 'None', None, None, None, None, None), ('230', None, '1340097569', 'None', None, None, None, None, None), ('230', None, '1340097570', 'None', None, None, None, None, None), ('230', None, '917849322', None, None, None, None, None, None)]
time used for this insertion : 0.12962770462036133
 save_final 

save missing photos in datou_result :
After save, about to update current !
 datou_cur_ids : []
 len(datou.list_steps) : 1
 output : {1340097567: ['917849322', 'temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg', []], 1340097569: ['917849322', 'temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg', []], 1340097570: ['917849322', 'temp/1740728611_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg', []]}
test rotate only is a success !
test rotate conditionnel
Inside batchDatouExec : verbose : True
##### chargement datou
SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=233

SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output,
mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt
WHERE mtdt.`id`=mtd.`type`
AND mtd.mtd_id=233
 
SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type
FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt
WHERE mtd.`type`=msid.`mtr_datou_step_type`
AND mtd.mtd_id= 233
AND msid.data_type=mdsdt.id
                    

SELECT mts_id_output, id_output, mts_id_input, id_input 
FROM MTRDatou.mtr_datou_step_by_step 
WHERE mtd_id=233
# VR 17-11-17 : to create in DB !
Here we check the datou graph and we reorder steps !
 Tree builded and cycle checked, now we need to re-order the steps ! 
We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue
We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id)
All sons are already in current list !
All sons are already in current list !
 DONE and to test : checkNoCycle !
Here we check the consistency of inputs/outputs number between the given ones and the db !
eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering !
Number of inputs / outputs for each step checked !
Here we check the consistency of outputs/inputs types during steps connections
eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput !
DataTypes for each output/input checked !
no param json to modify
List Step Type Loaded in datou : thcl, argmax, rotate
list_input_json : []
##### fin chargement datou
##### chargement data
##### Call load_data_input : nb_thread : 5
origin

        SELECT photo_id, url
        FROM MTRBack.photos ph
        WHERE photo_id IN (917849322) 
 Found this number of photos: 1
##### Call download_photos : nb_thread : 5
begin to download photo : 917849322
download finish for photo 917849322
we have missing 0 photos in the step downloads : 
photo missing : []
try to delete the photos missing in DB
##### After download_photos 
length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1
##### After load_data_input 
time to download the photos : 0.1620175838470459
#### fin chargement data
Blocking on flush ?
No conitnuing 
About to test input to load 
we should then remove the video here, and this would fix the bug of datou_current ! 
Calling datou_exec

Inside datou_exec : verbose : True
number of steps :  3
step1:thcl
Fri Feb 28 08:43:35 2025
VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next
VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case
VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec 
VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec
After prepare type args : <class 'list'>

Here we display some param of map_info ! 
map_filenames : {'temp/1740728615_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322}
map_photo_id_path_extension : {917849322: {'path': 'temp/1740728615_3374456_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}}
map_subphoto_mainphoto : {}

Beginning of datou step Thcl !
multi_thcl or not :False
multi_thcl_cond or not :False
dic_thcl :  {'500': 1}
we are using the classfication for only one thcl
500


In convert_file_to_np l 337 : 1
 l343 1
l357 after caffe.io.load_image
dimension du image : (3, (2448, 3264, 3))
dimension displayed !
time to import caffe and check if the image exist :  0.00020074844360351562
time to convert the images to numpy array :  0.8780472278594971
total time to convert the images to numpy array :  0.8785386085510254
list photo_ids error:
[]
list photo_ids correct :
[917849322]
number of photos to traite : 1
try to delete the photos incorrect in DB

tagging for thcl : 500
To do loadFromThcl(), then load ParamDescType : thcl500
 get_desc_type_from_thcl : type of cat <class 'mtr.database_queries.classification_admin_tools.ClassificationMetaTools'>

            SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists,
            photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at,
            type_classification
            FROM MTRDatou.classification_theme  WHERE  `id` IN (500) 
 thcls : [{'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'}]
 thcl {'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'}
 Update svm_hashtag_type_desc : 3517
 SELECT * FROM MTRDatou.photo_desc_type_params  WHERE id in (3517)
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34))
To loadFromThcl() : net_3517
begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 2500
l 3632 free memory gpu now : 6
wait 20 seconds
l 3637 free memory gpu now : 6  max_wait_temp :  1 max_wait :  0
 SELECT * FROM MTRDatou.photo_desc_type_params  WHERE id in (3517)
FOUND : 1
 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34))
 param : <mtr.database_queries.descriptor_queries.ParamDescriptorType object at 0x7f03c7f8ef70>, param.caffemodel : orientation_carte_grise_all_2
None
 mean_file_type : 
 mean_file_path : 
 prototxt_file_path : 
 model : orientation_carte_grise_all_2
 Inside get_net  
 Inside get_net before cache_data_model 
model_param file didn't exist 
 Inside get_net before  CDM.load_model_par_type  
model_name : orientation_carte_grise_all_2
model_type : caffe
list file need :
['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt']
file exist in s3 :
['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt']
file manque in s3 : 
[]
local folder  : /data/models_weight/orientation_carte_grise_all_2


/data/models_weight/orientation_carte_grise_all_2/caffemodel
size_local : 537110520 size in s3 : 537110520
create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:17
caffemodel already exist and didn't need to update


/data/models_weight/orientation_carte_grise_all_2/deploy_conv_normal.prototxt
size_local : 4626 size in s3 : 4626
create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16
deploy_conv_normal.prototxt already exist and didn't need to update


/data/models_weight/orientation_carte_grise_all_2/deploy_fc.prototxt
size_local : 1130 size in s3 : 1130
create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16
deploy_fc.prototxt already exist and didn't need to update


/data/models_weight/orientation_carte_grise_all_2/deploy.prototxt
size_local : 5653 size in s3 : 5653
create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16
deploy.prototxt already exist and didn't need to update


/data/models_weight/orientation_carte_grise_all_2/mean.npy
size_local : 1572992 size in s3 : 1572992
create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:31
mean.npy already exist and didn't need to update


/data/models_weight/orientation_carte_grise_all_2/synset_words.txt
size_local : 159 size in s3 : 159
create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16
synset_words.txt already exist and didn't need to update
 Inside get_net after CDM.load_model_par_type  
  After if not only_with_local_cache:
/home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/
Here before set mode gpu
Doing nothing but we could set mode gpu
after set mode gpu
 prototxt_filename : /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt
 caffemodel_filename : /data/models_weight/orientation_carte_grise_all_2/caffemodel
now we set caffe to gpu mode


 before predict
begin to check gpu status 
inside check gpu memory
havn't enough memory gpu , need / 2500
l 3632 free memory gpu now : 6
wait 20 seconds
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0228 08:44:22.417492 3374456 syncedmem.cpp:78] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
Command terminated by signal 6
42.49user 27.41system 2:08:58elapsed 0%CPU (0avgtext+0avgdata 5917540maxresident)k
5574656inputs+22832outputs (32210major+3457275minor)pagefaults 0swaps