python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 215 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.11311936378479004 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Sat Mar 1 18:35:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 215 wait 20 seconds l 3637 free memory gpu now : 215 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-03-01 18:35:52.264649: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-01 18:35:52.291137: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-01 18:35:52.292708: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fd844000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-01 18:35:52.292774: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-01 18:35:52.296330: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-01 18:35:52.497880: W tensorflow/compiler/xla/service/platform_util.cc:210] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported: 11553341440 2025-03-01 18:35:52.498185: I tensorflow/compiler/jit/xla_gpu_device.cc:161] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA 2025-03-01 18:35:52.498918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-01 18:35:52.499315: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-01 18:35:52.501787: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-01 18:35:52.504000: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-01 18:35:52.504346: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-01 18:35:52.507676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-01 18:35:52.508816: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-01 18:35:52.514173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-01 18:35:52.515048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-01 18:35:52.515134: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 Inside mask_sub_process Inside mask_detect Exception in mask_detect : CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory we want to redo the detection max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 138 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.1720283031463623 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Sat Mar 1 18:35:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 215 wait 20 seconds l 3637 free memory gpu now : 215 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-01 18:36:16.471678: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-01 18:36:16.499035: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-01 18:36:16.501030: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fd844000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-01 18:36:16.501075: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-01 18:36:16.504457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-01 18:36:16.730582: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1ff0de20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-01 18:36:16.730630: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-01 18:36:16.731505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-01 18:36:16.731910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-01 18:36:16.735024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-01 18:36:16.737613: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-01 18:36:16.738080: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-01 18:36:16.740839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-01 18:36:16.741745: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-01 18:36:16.745531: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-01 18:36:16.746277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-01 18:36:16.746330: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-01 18:36:16.746742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-01 18:36:16.746765: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-01 18:36:16.746773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-01 18:36:16.750941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 61 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-03-01 18:36:17.428280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-01 18:36:17.428358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-01 18:36:17.428378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-01 18:36:17.428397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-01 18:36:17.428415: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-01 18:36:17.428432: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-01 18:36:17.428449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-01 18:36:17.428467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-01 18:36:17.429234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-01 18:36:17.430133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-01 18:36:17.430172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-01 18:36:17.430191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-01 18:36:17.430208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-01 18:36:17.430225: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-01 18:36:17.430242: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-01 18:36:17.430259: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-01 18:36:17.430277: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-01 18:36:17.431047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-01 18:36:17.431080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-01 18:36:17.431090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-01 18:36:17.431099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-01 18:36:17.431910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 61 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2025-03-01 18:36:17.722532: F ./tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: out of memory max_time_sub_proc : 3600 Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 ERROR : 'int' object is not subscriptable reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: "'int' object is not subscriptable" /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_7b90246f7285e983362b27a932a2f1f627b3aad9 SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_7b90246f7285e983362b27a932a2f1f627b3aad9','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/March/01032025/python_test3//data_2/data_log/job/2025/March/01032025/python_test3/log-python3----short_python3--v--marlene-18:35:02.txt','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 1.9953036308288574 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Sat Mar 1 19:36:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740854176_2741006_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1740854176_2741006_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 (640, 960, 3) time for calcul the mask position with numpy : 0.0020225048065185547 nb_pixel_total : 5784 time to create 1 rle with old method : 0.0067768096923828125 time for calcul the mask position with numpy : 0.001476287841796875 nb_pixel_total : 13950 time to create 1 rle with old method : 0.015689849853515625 time for calcul the mask position with numpy : 0.0015761852264404297 nb_pixel_total : 16461 time to create 1 rle with old method : 0.01792144775390625 time for calcul the mask position with numpy : 0.0020112991333007812 nb_pixel_total : 38818 time to create 1 rle with old method : 0.04321002960205078 time for calcul the mask position with numpy : 0.00145721435546875 nb_pixel_total : 3930 time to create 1 rle with old method : 0.004676342010498047 time for calcul the mask position with numpy : 0.0014674663543701172 nb_pixel_total : 5629 time to create 1 rle with old method : 0.006699562072753906 time for calcul the mask position with numpy : 0.002341032028198242 nb_pixel_total : 84253 time to create 1 rle with old method : 0.08936476707458496 time for calcul the mask position with numpy : 0.00458526611328125 nb_pixel_total : 5463 time to create 1 rle with old method : 0.0063419342041015625 time for calcul the mask position with numpy : 0.0013720989227294922 nb_pixel_total : 5329 time to create 1 rle with old method : 0.0062940120697021484 time for calcul the mask position with numpy : 0.001440286636352539 nb_pixel_total : 3802 time to create 1 rle with old method : 0.004258871078491211 time for calcul the mask position with numpy : 0.0014324188232421875 nb_pixel_total : 2750 time to create 1 rle with old method : 0.0033223628997802734 time for calcul the mask position with numpy : 0.0014646053314208984 nb_pixel_total : 2937 time to create 1 rle with old method : 0.0034618377685546875 time for calcul the mask position with numpy : 0.0016872882843017578 nb_pixel_total : 29350 time to create 1 rle with old method : 0.039186954498291016 time for calcul the mask position with numpy : 0.0013735294342041016 nb_pixel_total : 1220 time to create 1 rle with old method : 0.0013730525970458984 time for calcul the mask position with numpy : 0.0013015270233154297 nb_pixel_total : 2452 time to create 1 rle with old method : 0.0026941299438476562 time for calcul the mask position with numpy : 0.001354217529296875 nb_pixel_total : 10812 time to create 1 rle with old method : 0.011647462844848633 time for calcul the mask position with numpy : 0.0014979839324951172 nb_pixel_total : 4268 time to create 1 rle with old method : 0.004848957061767578 time for calcul the mask position with numpy : 0.001344919204711914 nb_pixel_total : 6636 time to create 1 rle with old method : 0.007651805877685547 time for calcul the mask position with numpy : 0.0014111995697021484 nb_pixel_total : 598 time to create 1 rle with old method : 0.0007779598236083984 time for calcul the mask position with numpy : 0.0015053749084472656 nb_pixel_total : 16299 time to create 1 rle with old method : 0.018067121505737305 time for calcul the mask position with numpy : 0.0014984607696533203 nb_pixel_total : 2081 time to create 1 rle with old method : 0.002462625503540039 time for calcul the mask position with numpy : 0.0015125274658203125 nb_pixel_total : 7639 time to create 1 rle with old method : 0.008930444717407227 time for calcul the mask position with numpy : 0.0015120506286621094 nb_pixel_total : 4331 time to create 1 rle with old method : 0.005724191665649414 time for calcul the mask position with numpy : 0.001470327377319336 nb_pixel_total : 2759 time to create 1 rle with old method : 0.003641843795776367 time for calcul the mask position with numpy : 0.0014786720275878906 nb_pixel_total : 1320 time to create 1 rle with old method : 0.0018947124481201172 time for calcul the mask position with numpy : 0.0014629364013671875 nb_pixel_total : 734 time to create 1 rle with old method : 0.0011174678802490234 time for calcul the mask position with numpy : 0.0015325546264648438 nb_pixel_total : 8582 time to create 1 rle with old method : 0.009686470031738281 time for calcul the mask position with numpy : 0.0014824867248535156 nb_pixel_total : 3530 time to create 1 rle with old method : 0.0041539669036865234 time for calcul the mask position with numpy : 0.0015573501586914062 nb_pixel_total : 9857 time to create 1 rle with old method : 0.011565446853637695 time for calcul the mask position with numpy : 0.0015535354614257812 nb_pixel_total : 7629 time to create 1 rle with old method : 0.00925445556640625 time for calcul the mask position with numpy : 0.0015153884887695312 nb_pixel_total : 940 time to create 1 rle with old method : 0.0013086795806884766 time for calcul the mask position with numpy : 0.0015494823455810547 nb_pixel_total : 12719 time to create 1 rle with old method : 0.015567302703857422 time for calcul the mask position with numpy : 0.0014886856079101562 nb_pixel_total : 2448 time to create 1 rle with old method : 0.002989053726196289 time for calcul the mask position with numpy : 0.0015418529510498047 nb_pixel_total : 2776 time to create 1 rle with old method : 0.0033257007598876953 time for calcul the mask position with numpy : 0.0017514228820800781 nb_pixel_total : 27789 time to create 1 rle with old method : 0.031670570373535156 time for calcul the mask position with numpy : 0.0015039443969726562 nb_pixel_total : 4101 time to create 1 rle with old method : 0.004817485809326172 time for calcul the mask position with numpy : 0.0014269351959228516 nb_pixel_total : 1031 time to create 1 rle with old method : 0.001268625259399414 time for calcul the mask position with numpy : 0.001422882080078125 nb_pixel_total : 3326 time to create 1 rle with old method : 0.0039348602294921875 time for calcul the mask position with numpy : 0.0014522075653076172 nb_pixel_total : 12054 time to create 1 rle with old method : 0.013693571090698242 time for calcul the mask position with numpy : 0.001428842544555664 nb_pixel_total : 1676 time to create 1 rle with old method : 0.002108335494995117 time for calcul the mask position with numpy : 0.0014753341674804688 nb_pixel_total : 10621 time to create 1 rle with old method : 0.01297760009765625 time for calcul the mask position with numpy : 0.0014295578002929688 nb_pixel_total : 1647 time to create 1 rle with old method : 0.0021338462829589844 time for calcul the mask position with numpy : 0.0014376640319824219 nb_pixel_total : 342 time to create 1 rle with old method : 0.0005176067352294922 time for calcul the mask position with numpy : 0.0014994144439697266 nb_pixel_total : 14718 time to create 1 rle with old method : 0.018097400665283203 time for calcul the mask position with numpy : 0.0014925003051757812 nb_pixel_total : 3852 time to create 1 rle with old method : 0.004801034927368164 time for calcul the mask position with numpy : 0.0014710426330566406 nb_pixel_total : 4168 time to create 1 rle with old method : 0.005036115646362305 time for calcul the mask position with numpy : 0.001443624496459961 nb_pixel_total : 1252 time to create 1 rle with old method : 0.0015175342559814453 time for calcul the mask position with numpy : 0.0014150142669677734 nb_pixel_total : 2376 time to create 1 rle with old method : 0.003070354461669922 time for calcul the mask position with numpy : 0.0014231204986572266 nb_pixel_total : 425 time to create 1 rle with old method : 0.0005812644958496094 time for calcul the mask position with numpy : 0.0014395713806152344 nb_pixel_total : 874 time to create 1 rle with old method : 0.001224517822265625 time for calcul the mask position with numpy : 0.0015175342559814453 nb_pixel_total : 919 time to create 1 rle with old method : 0.00128936767578125 time for calcul the mask position with numpy : 0.0013430118560791016 nb_pixel_total : 1192 time to create 1 rle with old method : 0.0015087127685546875 time for calcul the mask position with numpy : 0.0014202594757080078 nb_pixel_total : 850 time to create 1 rle with old method : 0.0012044906616210938 time for calcul the mask position with numpy : 0.001440286636352539 nb_pixel_total : 2369 time to create 1 rle with old method : 0.0029916763305664062 time for calcul the mask position with numpy : 0.0014548301696777344 nb_pixel_total : 577 time to create 1 rle with old method : 0.0008418560028076172 time for calcul the mask position with numpy : 0.0014719963073730469 nb_pixel_total : 2206 time to create 1 rle with old method : 0.002761363983154297 time for calcul the mask position with numpy : 0.0014612674713134766 nb_pixel_total : 2407 time to create 1 rle with old method : 0.0032248497009277344 time for calcul the mask position with numpy : 0.0014345645904541016 nb_pixel_total : 1712 time to create 1 rle with old method : 0.0021910667419433594 time for calcul the mask position with numpy : 0.001421213150024414 nb_pixel_total : 337 time to create 1 rle with old method : 0.0004799365997314453 time for calcul the mask position with numpy : 0.0013000965118408203 nb_pixel_total : 693 time to create 1 rle with old method : 0.0009438991546630859 time for calcul the mask position with numpy : 0.0013568401336669922 nb_pixel_total : 885 time to create 1 rle with old method : 0.0010759830474853516 time for calcul the mask position with numpy : 0.001402139663696289 nb_pixel_total : 1489 time to create 1 rle with old method : 0.0020401477813720703 time for calcul the mask position with numpy : 0.0014700889587402344 nb_pixel_total : 1202 time to create 1 rle with old method : 0.00150299072265625 time for calcul the mask position with numpy : 0.0013439655303955078 nb_pixel_total : 592 time to create 1 rle with old method : 0.0007951259613037109 time for calcul the mask position with numpy : 0.0016989707946777344 nb_pixel_total : 1055 time to create 1 rle with old method : 0.001821279525756836 time for calcul the mask position with numpy : 0.0016241073608398438 nb_pixel_total : 8733 time to create 1 rle with old method : 0.01033639907836914 time for calcul the mask position with numpy : 0.0015268325805664062 nb_pixel_total : 3089 time to create 1 rle with old method : 0.0038886070251464844 time for calcul the mask position with numpy : 0.0014119148254394531 nb_pixel_total : 1092 time to create 1 rle with old method : 0.0016064643859863281 time for calcul the mask position with numpy : 0.0014307498931884766 nb_pixel_total : 8281 time to create 1 rle with old method : 0.009821653366088867 time for calcul the mask position with numpy : 0.0014884471893310547 nb_pixel_total : 298 time to create 1 rle with old method : 0.00047469139099121094 time for calcul the mask position with numpy : 0.0014793872833251953 nb_pixel_total : 16802 time to create 1 rle with old method : 0.020320892333984375 time for calcul the mask position with numpy : 0.0013175010681152344 nb_pixel_total : 1512 time to create 1 rle with old method : 0.0018107891082763672 time for calcul the mask position with numpy : 0.0014259815216064453 nb_pixel_total : 1438 time to create 1 rle with old method : 0.0017604827880859375 time for calcul the mask position with numpy : 0.0014159679412841797 nb_pixel_total : 9082 time to create 1 rle with old method : 0.010596990585327148 time for calcul the mask position with numpy : 0.0014071464538574219 nb_pixel_total : 1741 time to create 1 rle with old method : 0.0020148754119873047 time for calcul the mask position with numpy : 0.0015149116516113281 nb_pixel_total : 13205 time to create 1 rle with old method : 0.016785144805908203 time for calcul the mask position with numpy : 0.0014758110046386719 nb_pixel_total : 983 time to create 1 rle with old method : 0.00125885009765625 time for calcul the mask position with numpy : 0.0018057823181152344 nb_pixel_total : 39255 time to create 1 rle with old method : 0.04633975028991699 time for calcul the mask position with numpy : 0.001623392105102539 nb_pixel_total : 11109 time to create 1 rle with old method : 0.01260066032409668 time for calcul the mask position with numpy : 0.0014529228210449219 nb_pixel_total : 273 time to create 1 rle with old method : 0.0003643035888671875 time for calcul the mask position with numpy : 0.0014445781707763672 nb_pixel_total : 27801 time to create 1 rle with old method : 0.032450199127197266 time for calcul the mask position with numpy : 0.0014815330505371094 nb_pixel_total : 4154 time to create 1 rle with old method : 0.0049436092376708984 time for calcul the mask position with numpy : 0.0014598369598388672 nb_pixel_total : 970 time to create 1 rle with old method : 0.0011799335479736328 time for calcul the mask position with numpy : 0.0014424324035644531 nb_pixel_total : 3171 time to create 1 rle with old method : 0.003866434097290039 time for calcul the mask position with numpy : 0.0015680789947509766 nb_pixel_total : 9485 time to create 1 rle with old method : 0.011368036270141602 time for calcul the mask position with numpy : 0.0015230178833007812 nb_pixel_total : 615 time to create 1 rle with old method : 0.0008711814880371094 time for calcul the mask position with numpy : 0.0015671253204345703 nb_pixel_total : 18489 time to create 1 rle with old method : 0.02182173728942871 time for calcul the mask position with numpy : 0.001634836196899414 nb_pixel_total : 707 time to create 1 rle with old method : 0.0010728836059570312 time for calcul the mask position with numpy : 0.0013461112976074219 nb_pixel_total : 221 time to create 1 rle with old method : 0.00035953521728515625 time for calcul the mask position with numpy : 0.0014195442199707031 nb_pixel_total : 887 time to create 1 rle with old method : 0.0010895729064941406 time for calcul the mask position with numpy : 0.001461029052734375 nb_pixel_total : 1636 time to create 1 rle with old method : 0.002266407012939453 time for calcul the mask position with numpy : 0.0016446113586425781 nb_pixel_total : 7535 time to create 1 rle with old method : 0.009217262268066406 time for calcul the mask position with numpy : 0.0015132427215576172 nb_pixel_total : 601 time to create 1 rle with old method : 0.000885009765625 time for calcul the mask position with numpy : 0.0014455318450927734 nb_pixel_total : 634 time to create 1 rle with old method : 0.0008623600006103516 time for calcul the mask position with numpy : 0.0014410018920898438 nb_pixel_total : 1122 time to create 1 rle with old method : 0.0014951229095458984 time for calcul the mask position with numpy : 0.001508951187133789 nb_pixel_total : 913 time to create 1 rle with old method : 0.0014028549194335938 time for calcul the mask position with numpy : 0.0016884803771972656 nb_pixel_total : 492 time to create 1 rle with old method : 0.0007658004760742188 time for calcul the mask position with numpy : 0.0015048980712890625 nb_pixel_total : 1320 time to create 1 rle with old method : 0.0017895698547363281 time for calcul the mask position with numpy : 0.0014498233795166016 nb_pixel_total : 2195 time to create 1 rle with old method : 0.0028836727142333984 time for calcul the mask position with numpy : 0.0013952255249023438 nb_pixel_total : 945 time to create 1 rle with old method : 0.0011723041534423828 time for calcul the mask position with numpy : 0.0016193389892578125 nb_pixel_total : 1386 time to create 1 rle with old method : 0.0018069744110107422 time for calcul the mask position with numpy : 0.001474142074584961 nb_pixel_total : 885 time to create 1 rle with old method : 0.0012323856353759766 time for calcul the mask position with numpy : 0.0014681816101074219 nb_pixel_total : 9662 time to create 1 rle with old method : 0.011726140975952148 time for calcul the mask position with numpy : 0.0014755725860595703 nb_pixel_total : 476 time to create 1 rle with old method : 0.0006291866302490234 time for calcul the mask position with numpy : 0.0014591217041015625 nb_pixel_total : 1614 time to create 1 rle with old method : 0.0021507740020751953 insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) batch 1 Loaded 105 chid ids of type : 4677 Number RLEs to save : 9980 INSERT IGNORE INTO MTRPhoto.crop_segments (`crop_hashtag_id`, `x0`, `y0`, `length`) VALUES (%s, %s, %s , %s) first line : ('3691525227', '421', '60', '13') ... last line : ('3691525331', '280', '238', '18') INSERT IGNORE INTO MTRPhoto.crop_sum_segments (`crop_hashtag_id`, `sum_segments`) VALUES (%s, %s) TO DO : save crop sub photo not yet done ! After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : sam we use saveGeneral [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] Looping around the photos to save general results len do output : 1 /1189321094Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4573', None, None, None, None, None, None, None, None) ('4573', None, '1189321094', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', 'None', None, None, None, None, None)] time used for this insertion : 0.014193058013916016 save_final save missing photos in datou_result : time spend for datou_step_exec : 11.12224817276001 time spend to save output : 0.01446080207824707 total time spend for step 1 : 11.136708974838257 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'1189321094': [[, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], 'temp/1740854176_2741006_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png']} nb_objects detect : 105 ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.10668039321899414 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Sat Mar 1 19:36:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740854189_2741006_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1740854189_2741006_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] local folder : /data/models_weight/detection_plaque_valcor_010622 /data/models_weight/detection_plaque_valcor_010622/caffemodel size_local : 349723073 size in s3 : 349723073 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:56 caffemodel already exist and didn't need to update /data/models_weight/detection_plaque_valcor_010622/test.prototxt size_local : 7163 size in s3 : 7163 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:55 test.prototxt already exist and didn't need to update prototxt : /data/models_weight/detection_plaque_valcor_010622/test.prototxt caffemodel : /data/models_weight/detection_plaque_valcor_010622/caffemodel Loaded network /data/models_weight/detection_plaque_valcor_010622/caffemodel About to compute detect_faster_rcnn : len(args) : 1 Inside frcnn step exec : nb paths : 1 image_path : temp/1740854189_2741006_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg image_size (600, 800, 3) [[[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] ... [[ 14 16 16] [ 13 15 15] [ 11 13 13] ... [198 206 205] [198 206 205] [198 206 205]] [[ 16 18 18] [ 14 16 16] [ 11 13 13] ... [206 214 213] [206 214 213] [206 214 213]] [[ 13 15 15] [ 12 14 14] [ 9 11 11] ... [210 218 217] [210 218 217] [210 218 217]]] Detection took 0.071s for 300 object proposals c : plaque list_crops.shape (72, 5) proba : 0.06384326 (374.1269, 293.91873, 430.81, 317.80814) proba : 0.05221497 (382.17566, 297.18588, 552.35724, 344.65622) proba : 0.012271055 (345.35715, 272.42923, 468.85782, 320.7241) We are managing local photo_id len de result frcnn : 1 After datou_step_exec type output : time spend for datou_step_exec : 2.1746270656585693 time spend to save output : 0.0001285076141357422 total time spend for step 1 : 2.174755573272705 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True Inside saveFrcnn : final : True verbose : True threshold to save the result : 0.1 output flattener : [(0, 493029425, 4370, 374, 430, 293, 317, 0.06384326, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.05221497, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012271055, None)] Warning : no hashtag_ids to insert in the database final : True begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', '0', 0, '0', 493061979, '0', None)] time used for this insertion : 0.012819766998291016 [917754606] map_info['map_portfolio_photo'] : {} final : True mtd_id 4184 list_pids : [917754606] Looping around the photos to save general results len do output : 1 /0 before output type Managing all output in save final without adding information in the mtr_datou_result ('4184', None, None, None, None, None, None, None, None) ('4184', None, '917754606', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', None, None, None, None, None, None)] time used for this insertion : 0.01297450065612793 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {0: [[(0, 493029425, 4370, 374, 430, 293, 317, 0.06384326, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.05221497, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012271055, None)], 'temp/1740854189_2741006_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg']} ############################### TEST thcl ################################ TEST THCL Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=2 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=2 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 2 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=2 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 1 thcl is not linked in the step_by_step architecture ! WARNING : step 2 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (916235064) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 916235064 download finish for photo 916235064 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.2700364589691162 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:thcl Sat Mar 1 19:36:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740854192_2741006_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1740854192_2741006_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'355': 1} we are using the classfication for only one thcl 355 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (66, 66, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.006356477737426758 time to convert the images to numpy array : 0.0035178661346435547 total time to convert the images to numpy array : 0.010210037231445312 list photo_ids error: [] list photo_ids correct : [916235064] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 355 To do loadFromThcl(), then load ParamDescType : thcl355 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (355) thcls : [{'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3390 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) To loadFromThcl() : net_3390 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1428 wait 20 seconds l 3637 free memory gpu now : 1428 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) param : , param.caffemodel : car_360_1027 None mean_file_type : mean_file_path : prototxt_file_path : model : car_360_1027 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : car_360_1027 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/car_360_1027 /data/models_weight/car_360_1027/caffemodel size_local : 542944640 size in s3 : 542944640 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 caffemodel already exist and didn't need to update /data/models_weight/car_360_1027/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy_fc.prototxt size_local : 1132 size in s3 : 1132 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy.prototxt size_local : 5654 size in s3 : 5654 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/mean.npy size_local : 1572944 size in s3 : 1572944 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:55 mean.npy already exist and didn't need to update /data/models_weight/car_360_1027/synset_words.txt size_local : 13687 size in s3 : 13687 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/car_360_1027/deploy.prototxt caffemodel_filename : /data/models_weight/car_360_1027/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1428 wait 20 seconds l 3637 free memory gpu now : 1428 max_wait_temp : 1 max_wait : 0 dict_keys(['pool5', 'prob']) time used to do the prepocess of the images : 0.011014223098754883 time used to do the prediction : 0.18799209594726562 save descriptor for thcl : 355 (1, 512, 7, 7) Got the blobs of the net to insert : [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] code_as_byte_string:b'0000000000'| time to traite the descriptors : 0.046782493591308594 Testing : ['916235064'] In select_photos_meta_from_ids: SELECT photo_id, url, FROM_UNIXTIME(uploaded_at), latitude, longitude, text FROM MTRBack.photos WHERE photo_id IN (916235064) result : {916235064: {'photo_id': 916235064, 'url': 'https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/2017/10/14/6293d1bb790dc6902450e7c572b7d10b.jpg', 'latitude': 0.0, 'longitude': 0.0, 'text': None}} list_photo_exists : [916235064] storage_type for insertDescriptorsMulti : 1 To insert : 916235064 time to insert the descriptors : 0.6525740623474121 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : False verbose : True time used to find the portfolios of the photos select button_legend_list from MTRDatou.classification_theme where id = 355 SAVE THCL, output : {'916235064': [[('916235064', 'c_elysee_1027_gao__port_506302', 0.0018813931, 332, '355'), ('916235064', 'mokka_1027_gao__port_506374', 0.0011632781, 332, '355'), ('916235064', 'captur_1027_gao__port_506399', 0.00081577053, 332, '355'), ('916235064', 'sorento_1027_gao__port_506192', 0.0011774262, 332, '355'), ('916235064', 'navara_1027_gao__port_506205', 0.0025850278, 332, '355'), ('916235064', 'xc90_1027_gao__port_506350', 0.0041706343, 332, '355'), ('916235064', 'saxo_1027_gao__port_506052', 0.0034808815, 332, '355'), ('916235064', 'trafic_1027_gao__port_506295', 0.0073672812, 332, '355'), ('916235064', 'punto_evo_1027_gao__port_506066', 0.0021887328, 332, '355'), ('916235064', '5_1027_gao__port_506117', 0.0005798698, 332, '355'), ('916235064', '250_1027_gao__port_506065', 0.004590802, 332, '355'), ('916235064', 'd_max_1027_gao__port_506125', 0.003158864, 332, '355'), ('916235064', 'panamera_1027_gao__port_506387', 0.002250623, 332, '355'), ('916235064', 'alhambra_1027_gao__port_506381', 0.0053217937, 332, '355'), ('916235064', 'x6_1027_gao__port_506349', 0.001099889, 332, '355'), ('916235064', 'vitara_1027_gao__port_506328', 0.005401463, 332, '355'), ('916235064', 'fiesta_1027_gao__port_506377', 0.0039191553, 332, '355'), ('916235064', 'qashqai_1027_gao__port_506286', 0.0014788858, 332, '355'), ('916235064', '147_1027_gao__port_506124', 0.0019778302, 332, '355'), ('916235064', 'c5_1027_gao__port_506172', 0.0012440499, 332, '355'), ('916235064', 'q5_1027_gao__port_506206', 0.0015049254, 332, '355'), ('916235064', 'giulia_1027_gao__port_506178', 0.0021691038, 332, '355'), ('916235064', 'karl_1027_gao__port_506371', 0.0027082933, 332, '355'), ('916235064', 'mehari_1027_gao__port_506076', 0.0047014523, 332, '355'), ('916235064', '911_1027_gao__port_506114', 0.001941688, 332, '355'), ('916235064', '508_1027_gao__port_506329', 0.00095850835, 332, '355'), ('916235064', 'idea_1027_gao__port_506122', 0.0007700485, 332, '355'), ('916235064', 'megane_1027_gao__port_506220', 0.001946942, 332, '355'), ('916235064', 'ghibli_1027_gao__port_506174', 0.0013725797, 332, '355'), ('916235064', 'touareg_1027_gao__port_506224', 0.0016202665, 332, '355'), ('916235064', 'i10_1027_gao__port_506232', 0.0013925276, 332, '355'), ('916235064', 'jumper_1027_gao__port_506234', 0.010045014, 332, '355'), ('916235064', 'classe_clk_1027_gao__port_506173', 0.0010792105, 332, '355'), ('916235064', 'kuga_1027_gao__port_506181', 0.00084469665, 332, '355'), ('916235064', 'ct_1027_gao__port_506323', 0.0012519297, 332, '355'), ('916235064', 'leon_1027_gao__port_506326', 0.002584543, 332, '355'), ('916235064', 'ds5_1027_gao__port_506376', 0.0012428525, 332, '355'), ('916235064', 'cordoba_1027_gao__port_506048', 0.0028648772, 332, '355'), ('916235064', 'classe_cla_1027_gao__port_506400', 0.0012947723, 332, '355'), ('916235064', 'jumpy_1027_gao__port_506179', 0.010340144, 332, '355'), ('916235064', 'avensis_1027_gao__port_506311', 0.0018764758, 332, '355'), ('916235064', 'juke_1027_gao__port_506325', 0.0011344175, 332, '355'), ('916235064', '4008_1027_gao__port_506402', 0.0015759052, 332, '355'), ('916235064', '190_series_1027_gao__port_506051', 0.003980088, 332, '355'), ('916235064', 'serie_3_1027_gao__port_506294', 0.0028740154, 332, '355'), ('916235064', 'q7_1027_gao__port_506318', 0.0023356064, 332, '355'), ('916235064', 'glc_1027_gao__port_506303', 0.0012106791, 332, '355'), ('916235064', 'grand_vitara_1027_gao__port_506175', 0.0011447481, 332, '355'), ('916235064', 's40_1027_gao__port_506099', 0.002233547, 332, '355'), ('916235064', 'toledo_1027_gao__port_506061', 0.0017465212, 332, '355'), ('916235064', '5008_1027_gao__port_506337', 0.0047002463, 332, '355'), ('916235064', 'continental_1027_gao__port_506250', 0.0021915552, 332, '355'), ('916235064', 'coupe_1027_gao__port_506082', 0.0022625194, 332, '355'), ('916235064', 'iq_1027_gao__port_506166', 0.0018171659, 332, '355'), ('916235064', '407_1027_gao__port_506133', 0.0009057053, 332, '355'), ('916235064', 'touran_1027_gao__port_506308', 0.0020407252, 332, '355'), ('916235064', '300c_1027_gao__port_506078', 0.0025336228, 332, '355'), ('916235064', 'classe_gl_1027_gao__port_506340', 0.0044890386, 332, '355'), ('916235064', 'vivaro_1027_gao__port_506310', 0.0034253716, 332, '355'), ('916235064', 'sl_1027_gao__port_506100', 0.0031346534, 332, '355'), ('916235064', 'elise_1027_gao__port_506121', 0.0010253162, 332, '355'), ('916235064', '1007_1027_gao__port_506070', 0.0015355996, 332, '355'), ('916235064', 'i40_1027_gao__port_506218', 0.00059144926, 332, '355'), ('916235064', 'bipper_tepee_1027_gao__port_506227', 0.0040307343, 332, '355'), ('916235064', 'focus_1027_gao__port_506272', 0.001158534, 332, '355'), ('916235064', 'primera_1027_gao__port_506147', 0.0012156656, 332, '355'), ('916235064', 'r4_1027_gao__port_506160', 0.014968251, 332, '355'), ('916235064', 'a8_1027_gao__port_506265', 0.0011320371, 332, '355'), ('916235064', 'boxer_1027_gao__port_506202', 0.010545187, 332, '355'), ('916235064', 's5_1027_gao__port_506222', 0.0011982833, 332, '355'), ('916235064', 'r21_1027_gao__port_506093', 0.0041850894, 332, '355'), ('916235064', 'c3_1027_gao__port_506257', 0.002363449, 332, '355'), ('916235064', 'santa_fe_1027_gao__port_506208', 0.0016326198, 332, '355'), ('916235064', 'm4_1027_gao__port_506344', 0.0015565179, 332, '355'), ('916235064', 'safrane_1027_gao__port_506077', 0.0013958345, 332, '355'), ('916235064', 'classe_gle_1027_gao__port_506395', 0.0021976011, 332, '355'), ('916235064', '0_1027_gao__port_506094', 0.008827722, 332, '355'), ('916235064', 'ix35_1027_gao__port_506219', 0.0014616455, 332, '355'), ('916235064', 'carens_1027_gao__port_506298', 0.00088264124, 332, '355'), ('916235064', 'classe_a_1027_gao__port_506339', 0.0024712635, 332, '355'), ('916235064', 'ix20_1027_gao__port_506343', 0.0010093604, 332, '355'), ('916235064', 'note_1027_gao__port_506365', 0.0015963637, 332, '355'), ('916235064', 'a5_1027_gao__port_506200', 0.001532824, 332, '355'), ('916235064', 'sx4_1027_gao__port_506348', 0.0014917022, 332, '355'), ('916235064', 'sandero_1027_gao__port_506198', 0.0014586624, 332, '355'), ('916235064', '3008_1027_gao__port_506385', 0.0056449147, 332, '355'), ('916235064', 'q50_1027_gao__port_506239', 0.0011164558, 332, '355'), ('916235064', 'latitude_1027_gao__port_506236', 0.00080199086, 332, '355'), ('916235064', 'v40_1027_gao__port_506391', 0.0017144722, 332, '355'), ('916235064', 'xsara_1027_gao__port_506087', 0.0009823082, 332, '355'), ('916235064', 'grand_c_max_1027_gao__port_506342', 0.001795979, 332, '355'), ('916235064', 'swift_1027_gao__port_506149', 0.0015018844, 332, '355'), ('916235064', 'serie_1_1027_gao__port_506184', 0.0015138822, 332, '355'), ('916235064', 'xc70_1027_gao__port_506393', 0.003619721, 332, '355'), ('916235064', 'master_1027_gao__port_506203', 0.007957611, 332, '355'), ('916235064', 'clio_1027_gao__port_506280', 0.0029578775, 332, '355'), ('916235064', 'duster_1027_gao__port_506216', 0.0007445122, 332, '355'), ('916235064', 'traveller_1027_gao__port_506403', 0.004294637, 332, '355'), ('916235064', 'tipo_1027_gao__port_506355', 0.0010929342, 332, '355'), ('916235064', 'rav_4_1027_gao__port_506332', 0.0013603164, 332, '355'), ('916235064', 'coccinelle_1027_gao__port_506259', 0.0034951677, 332, '355'), ('916235064', 'spacetourer_1027_gao__port_506401', 0.0030973526, 332, '355'), ('916235064', 'xe_1027_gao__port_506357', 0.0014469317, 332, '355'), ('916235064', 'ds3_1027_gao__port_506324', 0.0013090447, 332, '355'), ('916235064', 'mx_5_1027_gao__port_506098', 0.002588615, 332, '355'), ('916235064', 'land_cruiser_1027_gao__port_506315', 0.009531714, 332, '355'), ('916235064', 'classe_b_1027_gao__port_506335', 0.0017213776, 332, '355'), ('916235064', '806_1027_gao__port_506088', 0.0025620162, 332, '355'), ('916235064', 'rx_8_1027_gao__port_506046', 0.003621402, 332, '355'), ('916235064', 'spark_1027_gao__port_506185', 0.0010076275, 332, '355'), ('916235064', '6_1027_gao__port_506171', 0.0011181086, 332, '355'), ('916235064', 'bravo_1027_gao__port_506080', 0.0014648947, 332, '355'), ('916235064', 'nx_1027_gao__port_506345', 0.0013679418, 332, '355'), ('916235064', 'sharan_1027_gao__port_506347', 0.005093381, 332, '355'), ('916235064', 'x_type_1027_gao__port_506067', 0.0007803996, 332, '355'), ('916235064', 'jimny_1027_gao__port_506233', 0.0046060546, 332, '355'), ('916235064', 'wrangler_1027_gao__port_506225', 0.0017995293, 332, '355'), ('916235064', 'c_crosser_1027_gao__port_506312', 0.0015930589, 332, '355'), ('916235064', 'v70_1027_gao__port_506278', 0.0019676976, 332, '355'), ('916235064', 'classe_e_1027_gao__port_506300', 0.0017368306, 332, '355'), ('916235064', 'classe_v_1027_gao__port_506258', 0.0127319945, 332, '355'), ('916235064', 'm3_1027_gao__port_506182', 0.0023371172, 332, '355'), ('916235064', 'abarth_500_1027_gao__port_506226', 0.004043662, 332, '355'), ('916235064', 'serie_6_1027_gao__port_506262', 0.0011314901, 332, '355'), ('916235064', 'modus_1027_gao__port_506146', 0.0018297221, 332, '355'), ('916235064', '3_1027_gao__port_506113', 0.0015081617, 332, '355'), ('916235064', '405_1027_gao__port_506108', 0.008055337, 332, '355'), ('916235064', 'allroad_1027_gao__port_506297', 0.0010597083, 332, '355'), ('916235064', 'auris_1027_gao__port_506322', 0.0011524343, 332, '355'), ('916235064', 'galaxy_1027_gao__port_506143', 0.00251524, 332, '355'), ('916235064', 'giulietta_1027_gao__port_506363', 0.00086513464, 332, '355'), ('916235064', '106_1027_gao__port_506073', 0.008268291, 332, '355'), ('916235064', 'classe_m_1027_gao__port_506154', 0.0030022666, 332, '355'), ('916235064', 'espace_1027_gao__port_506313', 0.0010648686, 332, '355'), ('916235064', 'panda_1027_gao__port_506189', 0.009030711, 332, '355'), ('916235064', 'rcz_1027_gao__port_506197', 0.0011292322, 332, '355'), ('916235064', '4007_1027_gao__port_506162', 0.00067938265, 332, '355'), ('916235064', 'classe_cl_1027_gao__port_506249', 0.0010857431, 332, '355'), ('916235064', 'leaf_1027_gao__port_506139', 0.0018039099, 332, '355'), ('916235064', 'octavia_1027_gao__port_506237', 0.0018598184, 332, '355'), ('916235064', 'ds4_1027_gao__port_506336', 0.0024157914, 332, '355'), ('916235064', 'freelander_1027_gao__port_506084', 0.002347627, 332, '355'), ('916235064', 'evasion_1027_gao__port_506109', 0.0031147455, 332, '355'), ('916235064', 'punto_1027_gao__port_506106', 0.0019493871, 332, '355'), ('916235064', '2cv_1027_gao__port_506045', 0.007973281, 332, '355'), ('916235064', 'x4_1027_gao__port_506392', 0.0017948691, 332, '355'), ('916235064', 'antara_1027_gao__port_506247', 0.0012468862, 332, '355'), ('916235064', 'murano_1027_gao__port_506316', 0.00060885405, 332, '355'), ('916235064', 'alto_1027_gao__port_506201', 0.009230469, 332, '355'), ('916235064', 'meriva_1027_gao__port_506353', 0.0013765879, 332, '355'), ('916235064', 'orlando_1027_gao__port_506305', 0.0018464964, 332, '355'), ('916235064', 'new_beetle_1027_gao__port_506050', 0.0011637734, 332, '355'), ('916235064', '306_1027_gao__port_506145', 0.0035064449, 332, '355'), ('916235064', 'tiguan_1027_gao__port_506362', 0.0026826242, 332, '355'), ('916235064', 's_type_1027_gao__port_506101', 0.0011382515, 332, '355'), ('916235064', 'c1_1027_gao__port_506128', 0.002751089, 332, '355'), ('916235064', 'vectra_1027_gao__port_506044', 0.0011989328, 332, '355'), ('916235064', 'outlander_1027_gao__port_506317', 0.0017121362, 332, '355'), ('916235064', '307_1027_gao__port_506074', 0.0020011098, 332, '355'), ('916235064', 'a6_s6_1027_gao__port_506134', 0.0016567372, 332, '355'), ('916235064', 'nemo_combi_1027_gao__port_506196', 0.0022666876, 332, '355'), ('916235064', 'berlingo_1027_gao__port_506194', 0.0046641463, 332, '355'), ('916235064', 'partner_1027_gao__port_506285', 0.0039417795, 332, '355'), ('916235064', 'cayenne_1027_gao__port_506177', 0.003798083, 332, '355'), ('916235064', 'quattroporte_1027_gao__port_506240', 0.0024442417, 332, '355'), ('916235064', 'c_max_1027_gao__port_506282', 0.0013124972, 332, '355'), ('916235064', 'fabia_1027_gao__port_506396', 0.005298366, 332, '355'), ('916235064', 'cx_3_1027_gao__port_506281', 0.0014460381, 332, '355'), ('916235064', 'x_trail_1027_gao__port_506264', 0.00183157, 332, '355'), ('916235064', 'scirocco_1027_gao__port_506276', 0.004790848, 332, '355'), ('916235064', 'matiz_1027_gao__port_506144', 0.0017560943, 332, '355'), ('916235064', 'tigra_1027_gao__port_506069', 0.0008540843, 332, '355'), ('916235064', 'escort_1027_gao__port_506091', 0.0048396573, 332, '355'), ('916235064', 'c2_1027_gao__port_506081', 0.001490254, 332, '355'), ('916235064', 'mini_1027_gao__port_506168', 0.0011921192, 332, '355'), ('916235064', 'i30_1027_gao__port_506291', 0.0006325544, 332, '355'), ('916235064', 'picanto_1027_gao__port_506238', 0.0029748597, 332, '355'), ('916235064', 'mito_1027_gao__port_506072', 0.0015076638, 332, '355'), ('916235064', 'impreza_1027_gao__port_506085', 0.0020189008, 332, '355'), ('916235064', 'kangoo_1027_gao__port_506235', 0.006586449, 332, '355'), ('916235064', 'a4_1027_gao__port_506193', 0.0019871974, 332, '355'), ('916235064', 'cayman_1027_gao__port_506268', 0.0018140591, 332, '355'), ('916235064', 'sportage_1027_gao__port_506148', 0.0014277643, 332, '355'), ('916235064', 'up_1027_gao__port_506356', 0.006862834, 332, '355'), ('916235064', 'optima_1027_gao__port_506386', 0.0008916809, 332, '355'), ('916235064', 'defender_1027_gao__port_506229', 0.0067219883, 332, '355'), ('916235064', 'serie_2_1027_gao__port_506256', 0.002227231, 332, '355'), ('916235064', 'edge_1027_gao__port_506187', 0.0008747772, 332, '355'), ('916235064', 'r19_1027_gao__port_506110', 0.0049417014, 332, '355'), ('916235064', 'jetta_1027_gao__port_506304', 0.0036195226, 332, '355'), ('916235064', 'eos_1027_gao__port_506115', 0.0038915256, 332, '355'), ('916235064', 'accord_1027_gao__port_506214', 0.0020126405, 332, '355'), ('916235064', 'yaris_1027_gao__port_506334', 0.0032337403, 332, '355'), ('916235064', 'classe_cls_1027_gao__port_506289', 0.0007851364, 332, '355'), ('916235064', 'polo_1027_gao__port_506361', 0.004310376, 332, '355'), ('916235064', 'serie_4_1027_gao__port_506366', 0.0011474979, 332, '355'), ('916235064', 'mini_cabriolet_1027_gao__port_506204', 0.0008377192, 332, '355'), ('916235064', 'prius_1027_gao__port_506190', 0.0011487924, 332, '355'), ('916235064', 'lodgy_1027_gao__port_506188', 0.002017542, 332, '355'), ('916235064', 'serie_7_1027_gao__port_506307', 0.0012478, 332, '355'), ('916235064', 'c15_1027_gao__port_506055', 0.01771071, 332, '355'), ('916235064', 'kadjar_1027_gao__port_506389', 0.0012505705, 332, '355'), ('916235064', 'insignia_1027_gao__port_506364', 0.0016434697, 332, '355'), ('916235064', '308_1027_gao__port_506279', 0.0021235575, 332, '355'), ('916235064', 'roomster_1027_gao__port_506241', 0.0018016031, 332, '355'), ('916235064', '80_1027_gao__port_506057', 0.004612555, 332, '355'), ('916235064', '309_1027_gao__port_506063', 0.013526515, 332, '355'), ('916235064', 'tucson_1027_gao__port_506320', 0.002123887, 332, '355'), ('916235064', 'x3_1027_gao__port_506212', 0.00089775614, 332, '355'), ('916235064', 'xf_1027_gao__port_506263', 0.0011165718, 332, '355'), ('916235064', '2008_1027_gao__port_506394', 0.0026411011, 332, '355'), ('916235064', 'passat_1027_gao__port_506306', 0.0014974466, 332, '355'), ('916235064', 'compass_1027_gao__port_506260', 0.003256633, 332, '355'), ('916235064', 'twingo_1027_gao__port_506309', 0.006499544, 332, '355'), ('916235064', 'micra_1027_gao__port_506221', 0.0035866385, 332, '355'), ('916235064', 'golf_1027_gao__port_506155', 0.0031929926, 332, '355'), ('916235064', 'soul_1027_gao__port_506176', 0.001289112, 332, '355'), ('916235064', 'rapid_1027_gao__port_506398', 0.0025911464, 332, '355'), ('916235064', 'forester_1027_gao__port_506360', 0.0022762169, 332, '355'), ('916235064', 'slk_1027_gao__port_506210', 0.001584437, 332, '355'), ('916235064', 'forfour_1027_gao__port_506341', 0.0021797004, 332, '355'), ('916235064', 'serie_5_1027_gao__port_506209', 0.0013694789, 332, '355'), ('916235064', 'xj_1027_gao__port_506170', 0.00260078, 332, '355'), ('916235064', 'pajero_1027_gao__port_506097', 0.00521105, 332, '355'), ('916235064', 'agila_1027_gao__port_506119', 0.004850626, 332, '355'), ('916235064', 'a6_1027_gao__port_506163', 0.001891025, 332, '355'), ('916235064', 'fox_1027_gao__port_506092', 0.0008464978, 332, '355'), ('916235064', 'boxster_1027_gao__port_506267', 0.0015941665, 332, '355'), ('916235064', 'altea_1027_gao__port_506246', 0.0021390924, 332, '355'), ('916235064', 'samurai_1027_gao__port_506047', 0.0062513477, 332, '355'), ('916235064', 'trax_1027_gao__port_506296', 0.0019438484, 332, '355'), ('916235064', 'getz_1027_gao__port_506058', 0.0016387689, 332, '355'), ('916235064', 'cherokee_1027_gao__port_506269', 0.0029811705, 332, '355'), ('916235064', 'koleos_1027_gao__port_506378', 0.0015590085, 332, '355'), ('916235064', 'z_series_1027_gao__port_506123', 0.0016559217, 332, '355'), ('916235064', 'ecosport_1027_gao__port_506271', 0.0013230023, 332, '355'), ('916235064', 'space_star_1027_gao__port_506277', 0.0021146275, 332, '355'), ('916235064', 'rs3_sportback_1027_gao__port_506207', 0.0019112906, 332, '355'), ('916235064', 'civic_1027_gao__port_506141', 0.0026895967, 332, '355'), ('916235064', 'talisman_1027_gao__port_506390', 0.00076136534, 332, '355'), ('916235064', 'f_pace_1027_gao__port_506314', 0.0016162562, 332, '355'), ('916235064', 'classe_c_1027_gao__port_506299', 0.0017939634, 332, '355'), ('916235064', 'tt_1027_gao__port_506075', 0.0013934163, 332, '355'), ('916235064', 'pathfinder_1027_gao__port_506183', 0.0016514763, 332, '355'), ('916235064', '156_1027_gao__port_506157', 0.0015444525, 332, '355'), ('916235064', 'cx_5_1027_gao__port_506228', 0.0014413258, 332, '355'), ('916235064', 'scenic_1027_gao__port_506255', 0.001608835, 332, '355'), ('916235064', 'yeti_1027_gao__port_506358', 0.0020913447, 332, '355'), ('916235064', 'mustang_1027_gao__port_506053', 0.010048273, 332, '355'), ('916235064', 'stilo_1027_gao__port_506060', 0.0010831231, 332, '355'), ('916235064', 'ateca_1027_gao__port_506382', 0.0017009291, 332, '355'), ('916235064', 'fiorino_1027_gao__port_506217', 0.009199812, 332, '355'), ('916235064', 'classe_glk_1027_gao__port_506290', 0.0017017137, 332, '355'), ('916235064', 'fortwo_1027_gao__port_506230', 0.0016010565, 332, '355'), ('916235064', 'cruze_1027_gao__port_506186', 0.0010052427, 332, '355'), ('916235064', '107_1027_gao__port_506213', 0.0016275876, 332, '355'), ('916235064', 'aygo_1027_gao__port_506248', 0.0032424442, 332, '355'), ('916235064', 'rx_1027_gao__port_506354', 0.0010631131, 332, '355'), ('916235064', '500_1027_gao__port_506245', 0.0016355745, 332, '355'), ('916235064', 'bora_1027_gao__port_506104', 0.0038163888, 332, '355'), ('916235064', 'transit_1027_gao__port_506111', 0.0048623825, 332, '355'), ('916235064', 'pt_cruiser_1027_gao__port_506054', 0.001916902, 332, '355'), ('916235064', 'patrol_1027_gao__port_506068', 0.0042399424, 332, '355'), ('916235064', 'r8_1027_gao__port_506156', 0.0012720374, 332, '355'), ('916235064', 'xm_1027_gao__port_506102', 0.0022674815, 332, '355'), ('916235064', 's60_1027_gao__port_506191', 0.003198801, 332, '355'), ('916235064', 'aveo_1027_gao__port_506158', 0.003840172, 332, '355'), ('916235064', 'captiva_1027_gao__port_506159', 0.0017190677, 332, '355'), ('916235064', 'ax_1027_gao__port_506153', 0.006894951, 332, '355'), ('916235064', 'rexton_1027_gao__port_506107', 0.0013023259, 332, '355'), ('916235064', 'camaro_1027_gao__port_506056', 0.0024898385, 332, '355'), ('916235064', 'ypsilon_1027_gao__port_506131', 0.0019545725, 332, '355'), ('916235064', 'delta_1027_gao__port_506165', 0.0014002884, 332, '355'), ('916235064', 'c4_1027_gao__port_506370', 0.00130077, 332, '355'), ('916235064', 'zx_1027_gao__port_506161', 0.0045930715, 332, '355'), ('916235064', 'verso_1027_gao__port_506242', 0.0007721163, 332, '355'), ('916235064', 'superb_1027_gao__port_506327', 0.0019939195, 332, '355'), ('916235064', 'r5_1027_gao__port_506253', 0.00954421, 332, '355'), ('916235064', 'caddy_1027_gao__port_506330', 0.013827651, 332, '355'), ('916235064', 'x5_1027_gao__port_506243', 0.0011203955, 332, '355'), ('916235064', 'f_type_1027_gao__port_506231', 0.0008298255, 332, '355'), ('916235064', 'fusion_1027_gao__port_506096', 0.0012671496, 332, '355'), ('916235064', 'dokker_1027_gao__port_506331', 0.0053591407, 332, '355'), ('916235064', '205_1027_gao__port_506062', 0.006682847, 332, '355'), ('916235064', 'macan_1027_gao__port_506195', 0.0015573405, 332, '355'), ('916235064', 'tourneo_1027_gao__port_506369', 0.0064033233, 332, '355'), ('916235064', '108_1027_gao__port_506384', 0.0052640997, 332, '355'), ('916235064', '9_3_1027_gao__port_506071', 0.00083740405, 332, '355'), ('916235064', 'mondeo_1027_gao__port_506116', 0.0014396363, 332, '355'), ('916235064', 'cr_v_1027_gao__port_506164', 0.0016411998, 332, '355'), ('916235064', 'c30_1027_gao__port_506090', 0.0017482885, 332, '355'), ('916235064', 'pulsar_1027_gao__port_506397', 0.0012018679, 332, '355'), ('916235064', 'ibiza_1027_gao__port_506273', 0.0037229387, 332, '355'), ('916235064', 'a1_1027_gao__port_506338', 0.0012346188, 332, '355'), ('916235064', 'matrix_1027_gao__port_506140', 0.0007078473, 332, '355'), ('916235064', 'carnival_1027_gao__port_506136', 0.002282125, 332, '355'), ('916235064', 'xantia_1027_gao__port_506086', 0.002195688, 332, '355'), ('916235064', 'terrano_1027_gao__port_506083', 0.002029213, 332, '355'), ('916235064', 'q3_1027_gao__port_506275', 0.0011265032, 332, '355'), ('916235064', 'hr_v_1027_gao__port_506283', 0.0017807676, 332, '355'), ('916235064', 'expert_1027_gao__port_506142', 0.0073703052, 332, '355'), ('916235064', 'multivan_1027_gao__port_506383', 0.0065056686, 332, '355'), ('916235064', 'venga_1027_gao__port_506380', 0.00080029847, 332, '355'), ('916235064', 'scudo_1027_gao__port_506129', 0.0055937483, 332, '355'), ('916235064', 'laguna_1027_gao__port_506368', 0.0007134827, 332, '355'), ('916235064', 'vel_satis_1027_gao__port_506130', 0.0027267812, 332, '355'), ('916235064', 'b_max_1027_gao__port_506367', 0.0017248096, 332, '355'), ('916235064', 'ignis_1027_gao__port_506292', 0.0043554124, 332, '355'), ('916235064', '159_1027_gao__port_506064', 0.0010783093, 332, '355'), ('916235064', 'grande_punto_1027_gao__port_506138', 0.0023633647, 332, '355'), ('916235064', 'logan_1027_gao__port_506167', 0.004397307, 332, '355'), ('916235064', 's_max_1027_gao__port_506223', 0.0012528104, 332, '355'), ('916235064', 'caravelle_1027_gao__port_506351', 0.0030297395, 332, '355'), ('916235064', 'adam_1027_gao__port_506079', 0.0010538534, 332, '355'), ('916235064', '406_1027_gao__port_506132', 0.001357302, 332, '355'), ('916235064', 'q30_1027_gao__port_506293', 0.0009716059, 332, '355'), ('916235064', 'almera_1027_gao__port_506089', 0.0010239484, 332, '355'), ('916235064', 'corsa_1027_gao__port_506095', 0.0025203766, 332, '355'), ('916235064', 'corolla_1027_gao__port_506120', 0.0026822493, 332, '355'), ('916235064', 'xc60_1027_gao__port_506388', 0.0018987572, 332, '355'), ('916235064', 'viano_1027_gao__port_506211', 0.0026947164, 332, '355'), ('916235064', 'pro_cee_d_1027_gao__port_506274', 0.000831932, 332, '355'), ('916235064', 'a3_1027_gao__port_506321', 0.0037381125, 332, '355'), ('916235064', 'v50_1027_gao__port_506150', 0.00079202396, 332, '355'), ('916235064', 'voyager_1027_gao__port_506169', 0.003053537, 332, '355'), ('916235064', 'corvette_1027_gao__port_506049', 0.003722441, 332, '355'), ('916235064', 'rio_1027_gao__port_506379', 0.0017739552, 332, '355'), ('916235064', 'jazz_1027_gao__port_506252', 0.0015306278, 332, '355'), ('916235064', '200_1027_gao__port_506112', 0.004086361, 332, '355'), ('916235064', 'tts_1027_gao__port_506199', 0.0011860583, 332, '355'), ('916235064', 'zafira_1027_gao__port_506287', 0.0026958042, 332, '355'), ('916235064', 'asx_1027_gao__port_506266', 0.0011406955, 332, '355'), ('916235064', '607_1027_gao__port_506118', 0.0012529608, 332, '355'), ('916235064', '207_1027_gao__port_506103', 0.0015147437, 332, '355'), ('916235064', 'classe_s_1027_gao__port_506301', 0.0031652288, 332, '355'), ('916235064', 'c6_1027_gao__port_506105', 0.0017347308, 332, '355'), ('916235064', 'express_1027_gao__port_506137', 0.016724404, 332, '355'), ('916235064', 'classe_gla_1027_gao__port_506352', 0.0018253444, 332, '355'), ('916235064', 'v60_1027_gao__port_506333', 0.0021458776, 332, '355'), ('916235064', 'ka_1027_gao__port_506180', 0.001415078, 332, '355'), ('916235064', 'range_rover_1027_gao__port_506254', 0.0020550597, 332, '355'), ('916235064', 'discovery_1027_gao__port_506375', 0.002297004, 332, '355'), ('916235064', 'classe_r_1027_gao__port_506270', 0.0013942738, 332, '355'), ('916235064', 'transporter_1027_gao__port_506319', 0.01197074, 332, '355'), ('916235064', 'cee_d_1027_gao__port_506288', 0.0010548707, 332, '355'), ('916235064', 'zoe_1027_gao__port_506244', 0.0020713028, 332, '355'), ('916235064', 'i20_1027_gao__port_506284', 0.0017868275, 332, '355'), ('916235064', 'gtv_1027_gao__port_506059', 0.005721702, 332, '355'), ('916235064', 's4_avant_1027_gao__port_506261', 0.002766034, 332, '355'), ('916235064', 'x1_1027_gao__port_506372', 0.0017141687, 332, '355'), ('916235064', 'autres_1027_gao__port_506127', 0.0048254468, 332, '355'), ('916235064', '208_1027_gao__port_506359', 0.0018687685, 332, '355'), ('916235064', 'c8_1027_gao__port_506135', 0.0012582489, 332, '355'), ('916235064', 'astra_1027_gao__port_506215', 0.0012625585, 332, '355'), ('916235064', '2_1027_gao__port_506151', 0.0009245066, 332, '355'), ('916235064', 'doblo_1027_gao__port_506251', 0.007467139, 332, '355'), ('916235064', '807_1027_gao__port_506152', 0.00072910934, 332, '355'), ('916235064', '206_1027_gao__port_506126', 0.0010385347, 332, '355'), ('916235064', 'a7_1027_gao__port_506373', 0.00069108506, 332, '355'), ('916235064', 'renegade_1027_gao__port_506346', 0.0021419784, 332, '355')]]} begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 1.5020370483398438e-05 save missing photos in datou_result : time spend for datou_step_exec : 45.836944580078125 time spend to save output : 1.5147745609283447 total time spend for step 1 : 47.35171914100647 step2:argmax Sat Mar 1 19:37:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740854192_2741006_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1740854192_2741006_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step Argmax ! calculate argmax for thcl : 355 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True photo_id : 916235064 output[photo_id] : [('916235064', 'c15_1027_gao__port_506055', 0.01771071, 332, '355'), 'temp/1740854192_2741006_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg'] begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 1 insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) first line : ('916235064', '2049863950', '332') ... last line : ('916235064', '2049863950', '332') time used for this insertion : 0.007909297943115234 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 1 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 0.011973142623901367 len list_finale : 1, len picture : 1 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('2', None, '916235064', 'c15_1027_gao__port_506055', None, None, '2049863950', '0.01771071', None)] time used for this insertion : 0.012012481689453125 saving photo_ids in datou_result photo id not in port begin to insert list_values into mtr_datou_result : length of list_values in save_final : 0 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [] time used for this insertion : 3.5762786865234375e-06 save missing photos in datou_result : time spend for datou_step_exec : 0.00022673606872558594 time spend to save output : 0.03214597702026367 total time spend for step 2 : 0.03237271308898926 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : {'916235064': [('916235064', 'c15_1027_gao__port_506055', 0.01771071, 332, '355'), 'temp/1740854192_2741006_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg']} ############################### TEST tfhub2 ################################ TEST TFHUB2 ######################## test with use_multi_inputs=0 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4567 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4567 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4567 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4567 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12835 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12836 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171252784,1171252764,1171252487) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171252487 begin to download photo : 1171252764 begin to download photo : 1171252784 download finish for photo 1171252487 download finish for photo 1171252784 download finish for photo 1171252764 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.17660212516784668 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Sat Mar 1 19:37:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740854239_2741006_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg': 1171252487, 'temp/1740854239_2741006_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg': 1171252784, 'temp/1740854239_2741006_1171252764_29d5179a892cc50aadc9d67245534b59.jpg': 1171252764} map_photo_id_path_extension : {1171252487: {'path': 'temp/1740854239_2741006_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg', 'extension': 'jpg'}, 1171252784: {'path': 'temp/1740854239_2741006_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg', 'extension': 'jpg'}, 1171252764: {'path': 'temp/1740854239_2741006_1171252764_29d5179a892cc50aadc9d67245534b59.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3609': 1} we are using the classfication for only one thcl 3609 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds 2025-03-01 19:39:28.295140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-01 19:39:28.296249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-01 19:39:28.296426: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-01 19:39:28.296475: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-01 19:39:28.311879: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-01 19:39:28.311994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-01 19:39:28.316453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-01 19:39:28.318018: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-01 19:39:28.333494: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-01 19:39:28.334539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-01 19:39:28.335025: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-01 19:39:28.371130: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-01 19:39:28.373103: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fd5a8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-01 19:39:28.373155: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-01 19:39:28.377284: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x9600d0d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-01 19:39:28.377315: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-01 19:39:28.378219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-01 19:39:28.378350: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-01 19:39:28.378380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-01 19:39:28.378468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-01 19:39:28.378508: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-01 19:39:28.378555: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-01 19:39:28.378608: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-01 19:39:28.378661: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-01 19:39:28.379818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-01 19:39:28.379894: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-01 19:39:28.380325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-01 19:39:28.380346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-01 19:39:28.380364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-01 19:39:28.381396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 1428 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3609 To do loadFromThcl(), then load ParamDescType : thcl3609 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3609) thcls : [{'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'}] thcl {'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'} Update svm_hashtag_type_desc : 5832 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5832) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5832, 'tfhub_19_06_2023', 1280, 1280, 'tfhub_19_06_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 6, 19, 12, 55, 22), datetime.datetime(2023, 6, 19, 12, 55, 22)) model_name : tfhub_19_06_2023 model_param file didn't exist model_name : tfhub_19_06_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-03-01 19:39:35.112524: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-01 19:39:35.113117: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-01 19:39:35.113704: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-01 19:39:35.114241: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-01 19:39:35.114838: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.98G (2129957120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-01 19:39:35.115410: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.79G (1916961536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-01 19:39:35.115998: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.61G (1725265408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-01 19:39:35.116521: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.45G (1552738816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/tfhub_19_06_2023 /data/models_weight/tfhub_19_06_2023/Confusion_Matrix.png size_local : 57753 size in s3 : 57753 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_jrm.jpg size_local : 79724 size in s3 : 79724 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcm.jpg size_local : 83556 size in s3 : 83556 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcnc.jpg size_local : 74107 size in s3 : 74107 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pehd.jpg size_local : 72705 size in s3 : 72705 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_tapis_vide.jpg size_local : 70874 size in s3 : 70874 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 checkpoint already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216488 size in s3 : 216488 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:21 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_weights.h5 size_local : 16499144 size in s3 : 16499144 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:15 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 77, in create_tfhub_model hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module"), File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171252487, 1171252784, 1171252764] map_info['map_portfolio_photo'] : {} final : True mtd_id 4567 list_pids : [1171252487, 1171252784, 1171252764] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4567', None, '1171252487', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252784', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252764', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.021416187286376953 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 135.62926244735718 time spend to save output : 0.030046463012695312 total time spend for step 0 : 135.65930891036987 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171252784': [(1171252784, 'jrm', 0.9677492, 4674, '3609'), 'temp/1687511175_1882837_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg'], '1171252764': [(1171252764, 'jrm', 0.9853587, 4674, '3609'), 'temp/1687511175_1882837_1171252764_29d5179a892cc50aadc9d67245534b59.jpg'], '1171252487': [(1171252487, 'jrm', 0.9262757, 4674, '3609'), 'temp/1687511175_1882837_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg']} got : None ######################## test with use_multi_inputs=1 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4621 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4621 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4621 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4621 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12927 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12928 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171291875,1171275372,1171275314) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171275314 begin to download photo : 1171275372 begin to download photo : 1171291875 download finish for photo 1171275372 download finish for photo 1171275314 download finish for photo 1171291875 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.22150683403015137 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Sat Mar 1 19:39:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740854375_2741006_1171275372_76d81364ff7df843bff095f45c07ba35.jpg': 1171275372, 'temp/1740854375_2741006_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg': 1171275314, 'temp/1740854375_2741006_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg': 1171291875} map_photo_id_path_extension : {1171275372: {'path': 'temp/1740854375_2741006_1171275372_76d81364ff7df843bff095f45c07ba35.jpg', 'extension': 'jpg'}, 1171275314: {'path': 'temp/1740854375_2741006_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg', 'extension': 'jpg'}, 1171291875: {'path': 'temp/1740854375_2741006_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3655': 1} we are using the classfication for only one thcl 3655 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 8 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 8 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 8 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 8 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 8 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 8 wait 20 seconds l 3637 free memory gpu now : 8 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3655 To do loadFromThcl(), then load ParamDescType : thcl3655 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3655) thcls : [{'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'}] thcl {'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'} Update svm_hashtag_type_desc : 5862 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5862) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5862, 'tfhub_18_7_2023', 1280, 1280, 'tfhub_18_7_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 7, 18, 22, 46, 29), datetime.datetime(2023, 7, 18, 22, 46, 29)) model_name : tfhub_18_7_2023 model_param file didn't exist model_name : tfhub_18_7_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/tfhub_18_7_2023 /data/models_weight/tfhub_18_7_2023/Confusion_Matrix.png size_local : 54360 size in s3 : 54360 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:28 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_jrm.jpg size_local : 72583 size in s3 : 72583 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcm.jpg size_local : 81681 size in s3 : 81681 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcnc.jpg size_local : 79510 size in s3 : 79510 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pehd.jpg size_local : 59936 size in s3 : 59936 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_tapis_vide.jpg size_local : 78974 size in s3 : 78974 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 checkpoint already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216529 size in s3 : 216529 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279748 size in s3 : 32279748 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_weights.h5 size_local : 16500868 size in s3 : 16500868 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:18 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 62, in create_tfhub_model fe_layer = hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module", File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171275372, 1171275314, 1171291875] map_info['map_portfolio_photo'] : {} final : True mtd_id 4621 list_pids : [1171275372, 1171275314, 1171291875] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4621', None, '1171275372', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171275314', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171291875', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.03168058395385742 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 133.49448108673096 time spend to save output : 0.03216195106506348 total time spend for step 0 : 133.52664303779602 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171291875': [(1171291875, 'tapis_vide', 0.97062814, 4723, '3655'), 'temp/1691745841_1143057_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg'], '1171275372': [(1171275372, 'tapis_vide', 0.9674145, 4723, '3655'), 'temp/1691745841_1143057_1171275372_76d81364ff7df843bff095f45c07ba35.jpg'], '1171275314': [(1171275314, 'tapis_vide', 0.96509415, 4723, '3655'), 'temp/1691745841_1143057_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg']} got : None ERROR tfhub2 FAILED ############################### TEST ordonner ################################ To do loadFromThcl(), then load ParamDescType : thcl358 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (358) thcls : [{'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3392 ['FirstUploadExperveo_vignette__port_505674', 'CAR_EXTERIEUR_Roue__port_503398', 'FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486', 'FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465', 'CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198', 'CAR_EXTERIEUR_Face_avant_axe_droit__port_504451', 'CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235', 'FirstUploadExperveo_vin__port_505675', 'CAR_EXTERIEUR_cote_droite__port_504108', 'CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565', 'FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201', 'cartegrise_orientation__port_505064', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217', 'CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531', 'CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218', 'CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214', 'CAR_EXTERIEUR_Angle_avant_droit__port_504087', 'FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484', 'CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563', 'CAR_EXTERIEUR_Angle_arriere_droit__port_504160', 'CAR_EXTERIEUR_arriere__port_504184', 'CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562', 'INTERIEUR_Compteur_kilometrique__port_503644', 'CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494', 'CAR_EXTERIEUR_Angle_arriere_gauche__port_504170', 'CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226', 'CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202', 'CAR_EXTERIEUR_moteur__port_503704', 'FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487', 'CAR_INTERIEUR_siege_arriere_class_1__port_506551', 'CAR_EXTERIEUR_avant__port_504146', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215', 'CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225', 'CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564', 'FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'CAR_INTERIEUR_coffre__port_503412', 'FirstUploadExperveo_rouetranche__port_505677', 'UploadPhotoImmatBest_class_1__port_505051', 'CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532', 'CAR_EXTERIEUR_angle_avant_gauche__port_504098', 'CAR_EXTERIEUR_face_avant_axe_gauche__port_504236', 'CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540', 'CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233', 'CAR_EXTERIEUR_roue_de_secour__port_503763', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199', 'CAR_EXTERIEUR_cote_gauche__port_504017', 'CAR_INTERIEUR_avant_volant_class_1__port_506503', 'CAR_INTERIEUR_avant_volant_class_2__port_506504', 'CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'] 51 SELECT hashtag_id,hashtag FROM MTRBack.hashtags where hashtag in ('FirstUploadExperveo_vignette__port_505674','CAR_EXTERIEUR_Roue__port_503398','FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486','FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485','CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465','CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198','CAR_EXTERIEUR_Face_avant_axe_droit__port_504451','CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235','FirstUploadExperveo_vin__port_505675','CAR_EXTERIEUR_cote_droite__port_504108','CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565','FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483','CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201','cartegrise_orientation__port_505064','CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217','CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531','CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218','CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214','CAR_EXTERIEUR_Angle_avant_droit__port_504087','FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484','CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563','CAR_EXTERIEUR_Angle_arriere_droit__port_504160','CAR_EXTERIEUR_arriere__port_504184','CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562','INTERIEUR_Compteur_kilometrique__port_503644','CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494','CAR_EXTERIEUR_Angle_arriere_gauche__port_504170','CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226','CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202','CAR_EXTERIEUR_moteur__port_503704','FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487','CAR_INTERIEUR_siege_arriere_class_1__port_506551','CAR_EXTERIEUR_avant__port_504146','CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215','CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225','CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564','FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482','CAR_INTERIEUR_coffre__port_503412','FirstUploadExperveo_rouetranche__port_505677','UploadPhotoImmatBest_class_1__port_505051','CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532','CAR_EXTERIEUR_angle_avant_gauche__port_504098','CAR_EXTERIEUR_face_avant_axe_gauche__port_504236','CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540','CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233','CAR_EXTERIEUR_roue_de_secour__port_503763','CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199','CAR_EXTERIEUR_cote_gauche__port_504017','CAR_INTERIEUR_avant_volant_class_1__port_506503','CAR_INTERIEUR_avant_volant_class_2__port_506504','CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'); 51 dict_keys(['cartegrise_orientation__port_505064', 'car_exterieur_angle_arriere_droit_axe_arriere__port_504217', 'car_exterieur_angle_arriere_droit_axe_droit__port_504215', 'car_exterieur_angle_arriere_droit__port_504160', 'car_exterieur_angle_arriere_gauche_axe_arriere__port_504201', 'car_exterieur_angle_arriere_gauche_axe_gauche__port_504199', 'car_exterieur_angle_arriere_gauche__port_504170', 'car_exterieur_angle_avant_droit_axe_arriere__port_504226', 'car_exterieur_angle_avant_droit_axe_droit__port_504225', 'car_exterieur_angle_avant_droit__port_504087', 'car_exterieur_angle_avant_gauche_axe_avant__port_504235', 'car_exterieur_angle_avant_gauche_axe_gauche__port_504234', 'car_exterieur_angle_avant_gauche__port_504098', 'car_exterieur_arriere__port_504184', 'car_exterieur_avant__port_504146', 'car_exterieur_cote_droite__port_504108', 'car_exterieur_cote_droit_axe_arriere__port_504214', 'car_exterieur_cote_droit_axe_avant__port_504465', 'car_exterieur_cote_gauche_axe_arriere__port_504198', 'car_exterieur_cote_gauche_axe_avant__port_504233', 'car_exterieur_cote_gauche__port_504017', 'car_exterieur_face_arriere_axe_droit__port_504218', 'car_exterieur_face_arriere_axe_gauche__port_504202', 'car_exterieur_face_avant_axe_droit__port_504451', 'car_exterieur_face_avant_axe_gauche__port_504236', 'car_exterieur_moteur__port_503704', 'car_exterieur_roue_de_secour__port_503763', 'car_exterieur_roue__port_503398', 'car_interieur_avant_volant_class_1__port_506503', 'car_interieur_avant_volant_class_2__port_506504', 'car_interieur_avant_volant_class_6_boutonrond__port_506562', 'car_interieur_avant_volant_class_6_class_2__port_506563', 'car_interieur_avant_volant_class_6_ecrangrosplan__port_506564', 'car_interieur_avant_volant_class_6_levierdevitesse__port_506565', 'car_interieur_avant_vue-arriere_class_1__port_506531', 'car_interieur_avant_vue-arriere_class_2__port_506532', 'car_interieur_avant_vue_droite_habitacle_class_1__port_506540', 'car_interieur_avant_vue_gauche_habitacle_class_1__port_506494', 'car_interieur_coffre__port_503412', 'car_interieur_siege_arriere_class_1__port_506551', 'firstuploadexperveo_carrosseriegrosplan_carrosserie__port_506483', 'firstuploadexperveo_carrosseriegrosplan_class_6__port_506487', 'firstuploadexperveo_carrosseriegrosplan_morceauderoue__port_506484', 'firstuploadexperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'firstuploadexperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'firstuploadexperveo_carrosseriegrosplan_vindanslamoquette__port_506486', 'firstuploadexperveo_rouetranche__port_505677', 'firstuploadexperveo_vignette__port_505674', 'firstuploadexperveo_vin__port_505675', 'interieur_compteur_kilometrique__port_503644', 'uploadphotoimmatbest_class_1__port_505051']) select photo_hashtag_type from MTRDatou.classification_theme where id = 358 thcl : 358 photo_hashtag_type : 337 SELECT phi.hashtag_id , phi.photo_id FROM MTRBack.photo_hashtag_ids phi, MTRUser.mtr_portfolio_photos mp where phi.type = 337 and phi.photo_id = mp.mtr_photo_id and mp.mtr_portfolio_id =510365; {510365: [(917973295, 1), (917973297, 1), (917973302, 1), (917973293, 7), (917973296, 11), (917973300, 11), (917973286, 13), (917973289, 13), (917973301, 24), (917973285, 29), (917973290, 29), (917973299, 29), (917973304, 35), (917973287, 36), (917973298, 36), (917973305, 36), (917973292, 37), (917973291, 41), (917973303, 41), (917973294, 42), (917973288, 46)]} ############################### TEST rotate ################################ test rotate only Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=230 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=230 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 230 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=230 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.18280816078186035 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:rotate Sat Mar 1 19:41:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step_rotate ! We are in a linear step without datou_depend ! rotate photos of 90,180,270 degres batch 1 select photo_id, hashtag_id, `type`, x0, x1, y0, y1, score, id from MTRPhoto.crop_hashtag_ids where photo_id in ( 917849322) and `type` in (0) Loaded 0 chid ids of type : 0 SELECT crop_hashtag_id, points FROM MTRPhoto.crop_polygon_points WHERE crop_hashtag_id in () map_chi : {} photo_id in download_rotate_and_save : 917849322 list_chi_loc : 0 Use all angle ! Rotation of photo 917849322 of 90 degree temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 90 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] 90 [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] shrink_image : False image_rotate : image_path : temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 180 degree temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 180 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] 180 [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] shrink_image : False image_rotate : image_path : temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 270 degree temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 270 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] 270 [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] shrink_image : False image_rotate : image_path : temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg image_rotate.mode : RGB About to upload 3 photos upload in portfolio : 551782 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740854510_2741006 we have uploaded 3 photos in the portfolio 551782 time of upload the photos Elapsed time : 0.9823634624481201 map_filename_photo_id : 3 map_filename_photo_id : {'temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg': 1340423525, 'temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg': 1340423526, 'temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg': 1340423527} Len new_chis : 3 Len list_new_chi_with_photo_id : 0 of type : 0 list_new_chi_with_photo_id : [] After datou_step_exec type output : time spend for datou_step_exec : 1.2008442878723145 time spend to save output : 6.151199340820312e-05 total time spend for step 1 : 1.2009057998657227 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : rotate we use saveGeneral [917849322] map_info['map_portfolio_photo'] : {} final : True mtd_id 230 list_pids : [917849322] Looping around the photos to save general results len do output : 3 /1340423525Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340423526Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340423527Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('230', None, None, None, None, None, None, None, None) ('230', None, '917849322', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('230', None, '1340423525', 'None', None, None, None, None, None), ('230', None, '1340423526', 'None', None, None, None, None, None), ('230', None, '1340423527', 'None', None, None, None, None, None), ('230', None, '917849322', None, None, None, None, None, None)] time used for this insertion : 0.01274871826171875 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {1340423525: ['917849322', 'temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg', []], 1340423526: ['917849322', 'temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg', []], 1340423527: ['917849322', 'temp/1740854509_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg', []]} test rotate only is a success ! test rotate conditionnel Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=233 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=233 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 233 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=233 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax, rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.12978076934814453 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 3 step1:thcl Sat Mar 1 19:41:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1740854511_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1740854511_2741006_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'500': 1} we are using the classfication for only one thcl 500 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (2448, 3264, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.00021910667419433594 time to convert the images to numpy array : 1.657759666442871 total time to convert the images to numpy array : 1.6583802700042725 list photo_ids error: [] list photo_ids correct : [917849322] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 500 To do loadFromThcl(), then load ParamDescType : thcl500 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (500) thcls : [{'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'}] thcl {'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'} Update svm_hashtag_type_desc : 3517 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) To loadFromThcl() : net_3517 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 6 wait 20 seconds l 3637 free memory gpu now : 6 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) param : , param.caffemodel : orientation_carte_grise_all_2 None mean_file_type : mean_file_path : prototxt_file_path : model : orientation_carte_grise_all_2 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : orientation_carte_grise_all_2 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/orientation_carte_grise_all_2 /data/models_weight/orientation_carte_grise_all_2/caffemodel size_local : 537110520 size in s3 : 537110520 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:17 caffemodel already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_fc.prototxt size_local : 1130 size in s3 : 1130 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt size_local : 5653 size in s3 : 5653 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:31 mean.npy already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/synset_words.txt size_local : 159 size in s3 : 159 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt caffemodel_filename : /data/models_weight/orientation_carte_grise_all_2/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 6 wait 20 seconds WARNING: Logging before InitGoogleLogging() is written to STDERR F0301 19:42:41.466463 2741006 syncedmem.cpp:78] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 26.02user 22.39system 1:07:15elapsed 1%CPU (0avgtext+0avgdata 5347572maxresident)k 4979456inputs+21896outputs (17528major+2663491minor)pagefaults 0swaps