python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -C 2526005' -s traitement_3459 -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3586921 load datou : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 0 init_dummy_multi_datou is not linked in the step_by_step architecture ! WARNING : step 1294 init_dummy_multi_datou is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2526005'] with mtr_portfolio_ids : ['20043199'] and first list_photo_ids : [] new path : /proc/3586921/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score over limit max, limiting to limit_max 20 list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 20 ; length of list_pids : 20 ; length of list_args : 20 time to download the photos : 3.263577461242676 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 4 11:02:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-04 11:02:42.916826: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-04 11:02:42.943150: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-04 11:02:42.945343: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8be0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-04 11:02:42.945386: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-04 11:02:42.949176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-04 11:02:43.185057: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3950ccb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-04 11:02:43.185100: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-04 11:02:43.186486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 11:02:43.186864: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 11:02:43.189560: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 11:02:43.192066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 11:02:43.192509: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 11:02:43.195615: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 11:02:43.196839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 11:02:43.200844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 11:02:43.202280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 11:02:43.202340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 11:02:43.203110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 11:02:43.203127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 11:02:43.203148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 11:02:43.204519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-04 11:02:43.462959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 11:02:43.463061: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 11:02:43.463089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 11:02:43.463115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 11:02:43.463139: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 11:02:43.463163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 11:02:43.463187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 11:02:43.463211: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 11:02:43.464741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 11:02:43.465829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 11:02:43.465857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 11:02:43.465873: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 11:02:43.465887: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 11:02:43.465901: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 11:02:43.465915: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 11:02:43.465929: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 11:02:43.465943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 11:02:43.467236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 11:02:43.467265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 11:02:43.467273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 11:02:43.467281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 11:02:43.468609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-04 11:02:50.043430: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 11:02:50.198681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 20 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 147.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 133.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 147.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 146.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 Detection mask done ! Trying to reset tf kernel 3587064 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5485 tf kernel not reseted sub process len(results) : 20 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 20 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10774 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0009675025939941406 nb_pixel_total : 54284 time to create 1 rle with old method : 0.06107354164123535 length of segment : 531 time for calcul the mask position with numpy : 0.014623403549194336 nb_pixel_total : 886922 time to create 1 rle with new method : 0.033834218978881836 length of segment : 1690 time for calcul the mask position with numpy : 9.942054748535156e-05 nb_pixel_total : 2163 time to create 1 rle with old method : 0.002692699432373047 length of segment : 48 time for calcul the mask position with numpy : 0.0015723705291748047 nb_pixel_total : 91184 time to create 1 rle with old method : 0.09829878807067871 length of segment : 380 time for calcul the mask position with numpy : 0.0011510848999023438 nb_pixel_total : 21730 time to create 1 rle with old method : 0.025443315505981445 length of segment : 402 time for calcul the mask position with numpy : 0.0003952980041503906 nb_pixel_total : 25383 time to create 1 rle with old method : 0.028856515884399414 length of segment : 172 time for calcul the mask position with numpy : 0.00015306472778320312 nb_pixel_total : 8187 time to create 1 rle with old method : 0.010161876678466797 length of segment : 73 time for calcul the mask position with numpy : 0.0012500286102294922 nb_pixel_total : 63555 time to create 1 rle with old method : 0.06937742233276367 length of segment : 475 time for calcul the mask position with numpy : 0.0001201629638671875 nb_pixel_total : 4223 time to create 1 rle with old method : 0.005438089370727539 length of segment : 65 time for calcul the mask position with numpy : 0.0033643245697021484 nb_pixel_total : 219336 time to create 1 rle with new method : 0.010098934173583984 length of segment : 769 time for calcul the mask position with numpy : 0.0004980564117431641 nb_pixel_total : 30751 time to create 1 rle with old method : 0.03529000282287598 length of segment : 218 time for calcul the mask position with numpy : 0.007929563522338867 nb_pixel_total : 538916 time to create 1 rle with new method : 0.01591968536376953 length of segment : 1341 time for calcul the mask position with numpy : 7.414817810058594e-05 nb_pixel_total : 1525 time to create 1 rle with old method : 0.0019414424896240234 length of segment : 49 time for calcul the mask position with numpy : 0.000568389892578125 nb_pixel_total : 35199 time to create 1 rle with old method : 0.03958296775817871 length of segment : 223 time for calcul the mask position with numpy : 7.867813110351562e-05 nb_pixel_total : 2155 time to create 1 rle with old method : 0.0026917457580566406 length of segment : 65 time for calcul the mask position with numpy : 0.0004794597625732422 nb_pixel_total : 19781 time to create 1 rle with old method : 0.02223348617553711 length of segment : 330 time for calcul the mask position with numpy : 0.0005552768707275391 nb_pixel_total : 19920 time to create 1 rle with old method : 0.023186922073364258 length of segment : 178 time for calcul the mask position with numpy : 0.0033783912658691406 nb_pixel_total : 159784 time to create 1 rle with new method : 0.00601959228515625 length of segment : 417 time for calcul the mask position with numpy : 0.0001697540283203125 nb_pixel_total : 3593 time to create 1 rle with old method : 0.004493236541748047 length of segment : 75 time for calcul the mask position with numpy : 0.015445947647094727 nb_pixel_total : 661608 time to create 1 rle with new method : 0.024444580078125 length of segment : 1082 time for calcul the mask position with numpy : 0.0003361701965332031 nb_pixel_total : 13742 time to create 1 rle with old method : 0.02220463752746582 length of segment : 168 time for calcul the mask position with numpy : 0.0048792362213134766 nb_pixel_total : 370327 time to create 1 rle with new method : 0.011704683303833008 length of segment : 1332 time for calcul the mask position with numpy : 0.0019478797912597656 nb_pixel_total : 158462 time to create 1 rle with new method : 0.004338741302490234 length of segment : 405 time for calcul the mask position with numpy : 0.001775503158569336 nb_pixel_total : 93179 time to create 1 rle with old method : 0.10169506072998047 length of segment : 469 time for calcul the mask position with numpy : 0.011605978012084961 nb_pixel_total : 517732 time to create 1 rle with new method : 0.02471184730529785 length of segment : 1812 time for calcul the mask position with numpy : 0.0069332122802734375 nb_pixel_total : 353490 time to create 1 rle with new method : 0.01623392105102539 length of segment : 1191 time for calcul the mask position with numpy : 0.005009174346923828 nb_pixel_total : 377388 time to create 1 rle with new method : 0.0142822265625 length of segment : 1493 time for calcul the mask position with numpy : 0.0010123252868652344 nb_pixel_total : 43778 time to create 1 rle with old method : 0.0480504035949707 length of segment : 336 time for calcul the mask position with numpy : 0.0008232593536376953 nb_pixel_total : 22308 time to create 1 rle with old method : 0.024033308029174805 length of segment : 348 time for calcul the mask position with numpy : 0.02208733558654785 nb_pixel_total : 981137 time to create 1 rle with new method : 0.041321516036987305 length of segment : 1342 time for calcul the mask position with numpy : 0.0012652873992919922 nb_pixel_total : 82541 time to create 1 rle with old method : 0.09420442581176758 length of segment : 413 time for calcul the mask position with numpy : 0.00012493133544921875 nb_pixel_total : 4591 time to create 1 rle with old method : 0.005677938461303711 length of segment : 123 time for calcul the mask position with numpy : 0.0006563663482666016 nb_pixel_total : 51405 time to create 1 rle with old method : 0.05776357650756836 length of segment : 279 time for calcul the mask position with numpy : 0.004546642303466797 nb_pixel_total : 310272 time to create 1 rle with new method : 0.008368730545043945 length of segment : 931 time for calcul the mask position with numpy : 0.0031685829162597656 nb_pixel_total : 198949 time to create 1 rle with new method : 0.007821321487426758 length of segment : 981 time for calcul the mask position with numpy : 0.01604485511779785 nb_pixel_total : 826465 time to create 1 rle with new method : 0.04083418846130371 length of segment : 1707 time for calcul the mask position with numpy : 0.0008707046508789062 nb_pixel_total : 41208 time to create 1 rle with old method : 0.047864675521850586 length of segment : 189 time for calcul the mask position with numpy : 0.0003104209899902344 nb_pixel_total : 11596 time to create 1 rle with old method : 0.013511896133422852 length of segment : 95 time for calcul the mask position with numpy : 0.0001499652862548828 nb_pixel_total : 2712 time to create 1 rle with old method : 0.003307819366455078 length of segment : 57 time for calcul the mask position with numpy : 0.0002644062042236328 nb_pixel_total : 5903 time to create 1 rle with old method : 0.006799936294555664 length of segment : 130 time for calcul the mask position with numpy : 0.0004954338073730469 nb_pixel_total : 19522 time to create 1 rle with old method : 0.02282857894897461 length of segment : 220 time for calcul the mask position with numpy : 0.0007793903350830078 nb_pixel_total : 30808 time to create 1 rle with old method : 0.03476715087890625 length of segment : 279 time for calcul the mask position with numpy : 0.012656450271606445 nb_pixel_total : 762485 time to create 1 rle with new method : 0.029107332229614258 length of segment : 1002 time for calcul the mask position with numpy : 0.004265546798706055 nb_pixel_total : 367885 time to create 1 rle with new method : 0.01064300537109375 length of segment : 996 time for calcul the mask position with numpy : 0.00026488304138183594 nb_pixel_total : 16085 time to create 1 rle with old method : 0.018757104873657227 length of segment : 133 time for calcul the mask position with numpy : 0.0006885528564453125 nb_pixel_total : 21787 time to create 1 rle with old method : 0.02571272850036621 length of segment : 227 time for calcul the mask position with numpy : 0.0003807544708251953 nb_pixel_total : 9197 time to create 1 rle with old method : 0.010923147201538086 length of segment : 202 time for calcul the mask position with numpy : 0.013925552368164062 nb_pixel_total : 792790 time to create 1 rle with new method : 0.023610830307006836 length of segment : 1269 time for calcul the mask position with numpy : 0.0010380744934082031 nb_pixel_total : 63436 time to create 1 rle with old method : 0.07536602020263672 length of segment : 442 time for calcul the mask position with numpy : 0.0015392303466796875 nb_pixel_total : 103139 time to create 1 rle with old method : 0.11884665489196777 length of segment : 474 time for calcul the mask position with numpy : 0.003790140151977539 nb_pixel_total : 318161 time to create 1 rle with new method : 0.007967233657836914 length of segment : 918 time for calcul the mask position with numpy : 0.0001919269561767578 nb_pixel_total : 7623 time to create 1 rle with old method : 0.009164810180664062 length of segment : 162 time for calcul the mask position with numpy : 0.01622319221496582 nb_pixel_total : 748735 time to create 1 rle with new method : 0.02554178237915039 length of segment : 2312 time for calcul the mask position with numpy : 0.0003383159637451172 nb_pixel_total : 15220 time to create 1 rle with old method : 0.024863243103027344 length of segment : 176 time for calcul the mask position with numpy : 0.0002415180206298828 nb_pixel_total : 8357 time to create 1 rle with old method : 0.011243104934692383 length of segment : 130 time for calcul the mask position with numpy : 0.005443096160888672 nb_pixel_total : 454969 time to create 1 rle with new method : 0.012479066848754883 length of segment : 1923 time for calcul the mask position with numpy : 0.001621246337890625 nb_pixel_total : 121002 time to create 1 rle with old method : 0.13500571250915527 length of segment : 506 time for calcul the mask position with numpy : 0.0002090930938720703 nb_pixel_total : 2621 time to create 1 rle with old method : 0.0032775402069091797 length of segment : 78 time for calcul the mask position with numpy : 0.0002982616424560547 nb_pixel_total : 4418 time to create 1 rle with old method : 0.005366086959838867 length of segment : 129 time for calcul the mask position with numpy : 0.014542818069458008 nb_pixel_total : 572008 time to create 1 rle with new method : 0.028319597244262695 length of segment : 1332 time for calcul the mask position with numpy : 0.0010216236114501953 nb_pixel_total : 73376 time to create 1 rle with old method : 0.08290505409240723 length of segment : 559 time for calcul the mask position with numpy : 0.00016021728515625 nb_pixel_total : 7532 time to create 1 rle with old method : 0.008807897567749023 length of segment : 128 time for calcul the mask position with numpy : 0.0006911754608154297 nb_pixel_total : 54936 time to create 1 rle with old method : 0.06201982498168945 length of segment : 279 time for calcul the mask position with numpy : 0.0048596858978271484 nb_pixel_total : 371666 time to create 1 rle with new method : 0.011230230331420898 length of segment : 1054 time for calcul the mask position with numpy : 0.0013499259948730469 nb_pixel_total : 44460 time to create 1 rle with old method : 0.05151867866516113 length of segment : 145 time for calcul the mask position with numpy : 0.0009050369262695312 nb_pixel_total : 47848 time to create 1 rle with old method : 0.05810260772705078 length of segment : 144 time for calcul the mask position with numpy : 0.0001392364501953125 nb_pixel_total : 2455 time to create 1 rle with old method : 0.0030901432037353516 length of segment : 53 time for calcul the mask position with numpy : 0.015764713287353516 nb_pixel_total : 593624 time to create 1 rle with new method : 0.022646427154541016 length of segment : 1330 time for calcul the mask position with numpy : 0.0020263195037841797 nb_pixel_total : 124767 time to create 1 rle with old method : 0.1397874355316162 length of segment : 491 time for calcul the mask position with numpy : 0.00014638900756835938 nb_pixel_total : 5276 time to create 1 rle with old method : 0.006233930587768555 length of segment : 70 time for calcul the mask position with numpy : 0.00034809112548828125 nb_pixel_total : 16040 time to create 1 rle with old method : 0.01919388771057129 length of segment : 186 time for calcul the mask position with numpy : 0.00497889518737793 nb_pixel_total : 284966 time to create 1 rle with new method : 0.009333133697509766 length of segment : 998 time for calcul the mask position with numpy : 0.00021314620971679688 nb_pixel_total : 8882 time to create 1 rle with old method : 0.010465383529663086 length of segment : 112 time for calcul the mask position with numpy : 8.511543273925781e-05 nb_pixel_total : 2861 time to create 1 rle with old method : 0.0035552978515625 length of segment : 69 time for calcul the mask position with numpy : 6.67572021484375e-05 nb_pixel_total : 2886 time to create 1 rle with old method : 0.0036144256591796875 length of segment : 71 time for calcul the mask position with numpy : 0.00040531158447265625 nb_pixel_total : 14534 time to create 1 rle with old method : 0.016528606414794922 length of segment : 177 time for calcul the mask position with numpy : 0.0007383823394775391 nb_pixel_total : 18724 time to create 1 rle with old method : 0.021210432052612305 length of segment : 193 time for calcul the mask position with numpy : 0.004200935363769531 nb_pixel_total : 145052 time to create 1 rle with old method : 0.18637871742248535 length of segment : 1024 time for calcul the mask position with numpy : 0.0022199153900146484 nb_pixel_total : 91026 time to create 1 rle with old method : 0.09891200065612793 length of segment : 363 time for calcul the mask position with numpy : 0.0006923675537109375 nb_pixel_total : 44168 time to create 1 rle with old method : 0.049547433853149414 length of segment : 192 time for calcul the mask position with numpy : 0.0004904270172119141 nb_pixel_total : 30100 time to create 1 rle with old method : 0.03435969352722168 length of segment : 179 time for calcul the mask position with numpy : 0.0005893707275390625 nb_pixel_total : 36390 time to create 1 rle with old method : 0.041634321212768555 length of segment : 196 time for calcul the mask position with numpy : 0.00046133995056152344 nb_pixel_total : 26447 time to create 1 rle with old method : 0.029989004135131836 length of segment : 280 time for calcul the mask position with numpy : 0.00035762786865234375 nb_pixel_total : 26733 time to create 1 rle with old method : 0.02999258041381836 length of segment : 314 time for calcul the mask position with numpy : 0.0013725757598876953 nb_pixel_total : 57703 time to create 1 rle with old method : 0.06282687187194824 length of segment : 351 time for calcul the mask position with numpy : 0.0015211105346679688 nb_pixel_total : 111166 time to create 1 rle with old method : 0.12415671348571777 length of segment : 476 time for calcul the mask position with numpy : 0.022751569747924805 nb_pixel_total : 866957 time to create 1 rle with new method : 0.03405499458312988 length of segment : 1272 time for calcul the mask position with numpy : 0.002274751663208008 nb_pixel_total : 137638 time to create 1 rle with old method : 0.17499470710754395 length of segment : 816 time for calcul the mask position with numpy : 0.0008306503295898438 nb_pixel_total : 53832 time to create 1 rle with old method : 0.0603787899017334 length of segment : 266 time for calcul the mask position with numpy : 0.0008199214935302734 nb_pixel_total : 41312 time to create 1 rle with old method : 0.0477752685546875 length of segment : 177 time for calcul the mask position with numpy : 0.0037288665771484375 nb_pixel_total : 311959 time to create 1 rle with new method : 0.008752584457397461 length of segment : 913 time for calcul the mask position with numpy : 0.0006246566772460938 nb_pixel_total : 37318 time to create 1 rle with old method : 0.042768001556396484 length of segment : 358 time for calcul the mask position with numpy : 0.0001804828643798828 nb_pixel_total : 9394 time to create 1 rle with old method : 0.011319875717163086 length of segment : 110 time for calcul the mask position with numpy : 0.0005776882171630859 nb_pixel_total : 42268 time to create 1 rle with old method : 0.0480043888092041 length of segment : 178 time for calcul the mask position with numpy : 0.0006287097930908203 nb_pixel_total : 47823 time to create 1 rle with old method : 0.05197572708129883 length of segment : 331 time for calcul the mask position with numpy : 0.009830713272094727 nb_pixel_total : 475334 time to create 1 rle with new method : 0.019163131713867188 length of segment : 1148 time for calcul the mask position with numpy : 0.0016658306121826172 nb_pixel_total : 115760 time to create 1 rle with old method : 0.12578773498535156 length of segment : 594 time for calcul the mask position with numpy : 0.0008068084716796875 nb_pixel_total : 47468 time to create 1 rle with old method : 0.053067922592163086 length of segment : 205 time for calcul the mask position with numpy : 0.00028443336486816406 nb_pixel_total : 12853 time to create 1 rle with old method : 0.01480555534362793 length of segment : 150 time for calcul the mask position with numpy : 0.0001392364501953125 nb_pixel_total : 6432 time to create 1 rle with old method : 0.007494211196899414 length of segment : 132 time for calcul the mask position with numpy : 0.0003173351287841797 nb_pixel_total : 22591 time to create 1 rle with old method : 0.025952577590942383 length of segment : 151 time for calcul the mask position with numpy : 0.0014133453369140625 nb_pixel_total : 61822 time to create 1 rle with old method : 0.0803225040435791 length of segment : 445 time for calcul the mask position with numpy : 0.013854026794433594 nb_pixel_total : 729937 time to create 1 rle with new method : 0.02828073501586914 length of segment : 1687 time for calcul the mask position with numpy : 0.00012683868408203125 nb_pixel_total : 4798 time to create 1 rle with old method : 0.0054433345794677734 length of segment : 105 time for calcul the mask position with numpy : 0.005125999450683594 nb_pixel_total : 301212 time to create 1 rle with new method : 0.011373281478881836 length of segment : 936 time for calcul the mask position with numpy : 0.0002288818359375 nb_pixel_total : 7332 time to create 1 rle with old method : 0.008296012878417969 length of segment : 123 time for calcul the mask position with numpy : 0.0012121200561523438 nb_pixel_total : 37481 time to create 1 rle with old method : 0.0420994758605957 length of segment : 467 time for calcul the mask position with numpy : 0.00020647048950195312 nb_pixel_total : 4720 time to create 1 rle with old method : 0.005730867385864258 length of segment : 95 time for calcul the mask position with numpy : 0.00023818016052246094 nb_pixel_total : 5459 time to create 1 rle with old method : 0.006686210632324219 length of segment : 110 time for calcul the mask position with numpy : 0.011835813522338867 nb_pixel_total : 537947 time to create 1 rle with new method : 0.020424842834472656 length of segment : 1836 time for calcul the mask position with numpy : 9.894371032714844e-05 nb_pixel_total : 1562 time to create 1 rle with old method : 0.002186298370361328 length of segment : 38 time for calcul the mask position with numpy : 0.0006384849548339844 nb_pixel_total : 49049 time to create 1 rle with old method : 0.054991960525512695 length of segment : 184 time for calcul the mask position with numpy : 0.00017762184143066406 nb_pixel_total : 6642 time to create 1 rle with old method : 0.008101463317871094 length of segment : 88 time for calcul the mask position with numpy : 0.0005407333374023438 nb_pixel_total : 26083 time to create 1 rle with old method : 0.02983379364013672 length of segment : 329 time for calcul the mask position with numpy : 0.0002014636993408203 nb_pixel_total : 10767 time to create 1 rle with old method : 0.012922525405883789 length of segment : 111 time for calcul the mask position with numpy : 0.00020933151245117188 nb_pixel_total : 8050 time to create 1 rle with old method : 0.010274887084960938 length of segment : 94 time for calcul the mask position with numpy : 0.00024771690368652344 nb_pixel_total : 6006 time to create 1 rle with old method : 0.007082223892211914 length of segment : 141 time for calcul the mask position with numpy : 0.008090496063232422 nb_pixel_total : 390955 time to create 1 rle with new method : 0.01496744155883789 length of segment : 1153 time for calcul the mask position with numpy : 0.00032591819763183594 nb_pixel_total : 13664 time to create 1 rle with old method : 0.015367984771728516 length of segment : 121 time for calcul the mask position with numpy : 0.0006382465362548828 nb_pixel_total : 32647 time to create 1 rle with old method : 0.03803110122680664 length of segment : 217 time for calcul the mask position with numpy : 0.0002315044403076172 nb_pixel_total : 12845 time to create 1 rle with old method : 0.015382051467895508 length of segment : 126 time for calcul the mask position with numpy : 0.0006949901580810547 nb_pixel_total : 32928 time to create 1 rle with old method : 0.039202213287353516 length of segment : 197 time for calcul the mask position with numpy : 0.008031368255615234 nb_pixel_total : 346827 time to create 1 rle with new method : 0.014745235443115234 length of segment : 1184 time for calcul the mask position with numpy : 0.0010190010070800781 nb_pixel_total : 46510 time to create 1 rle with old method : 0.05154776573181152 length of segment : 486 time for calcul the mask position with numpy : 0.0031595230102539062 nb_pixel_total : 200932 time to create 1 rle with new method : 0.005877256393432617 length of segment : 753 time for calcul the mask position with numpy : 0.002062082290649414 nb_pixel_total : 50260 time to create 1 rle with old method : 0.056253671646118164 length of segment : 654 time for calcul the mask position with numpy : 0.0009419918060302734 nb_pixel_total : 55061 time to create 1 rle with old method : 0.06206989288330078 length of segment : 172 time for calcul the mask position with numpy : 0.0014617443084716797 nb_pixel_total : 56103 time to create 1 rle with old method : 0.062335968017578125 length of segment : 453 time for calcul the mask position with numpy : 0.0009658336639404297 nb_pixel_total : 19898 time to create 1 rle with old method : 0.02239084243774414 length of segment : 373 time for calcul the mask position with numpy : 0.007820367813110352 nb_pixel_total : 399601 time to create 1 rle with new method : 0.01946854591369629 length of segment : 1601 time for calcul the mask position with numpy : 9.584426879882812e-05 nb_pixel_total : 1385 time to create 1 rle with old method : 0.0017936229705810547 length of segment : 56 time for calcul the mask position with numpy : 0.0003399848937988281 nb_pixel_total : 13106 time to create 1 rle with old method : 0.015903472900390625 length of segment : 91 time for calcul the mask position with numpy : 0.0012047290802001953 nb_pixel_total : 33978 time to create 1 rle with old method : 0.03873300552368164 length of segment : 417 time for calcul the mask position with numpy : 0.00046181678771972656 nb_pixel_total : 14631 time to create 1 rle with old method : 0.016989707946777344 length of segment : 143 time for calcul the mask position with numpy : 0.00917816162109375 nb_pixel_total : 448880 time to create 1 rle with new method : 0.019282817840576172 length of segment : 2611 time for calcul the mask position with numpy : 0.00039267539978027344 nb_pixel_total : 8202 time to create 1 rle with old method : 0.009627580642700195 length of segment : 83 time for calcul the mask position with numpy : 0.00044417381286621094 nb_pixel_total : 11532 time to create 1 rle with old method : 0.014040946960449219 length of segment : 150 time for calcul the mask position with numpy : 0.0001659393310546875 nb_pixel_total : 4881 time to create 1 rle with old method : 0.005724668502807617 length of segment : 74 time spent for convertir_results : 14.248728036880493 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 138 chid ids of type : 3663 Number RLEs to save : 69433 save missing photos in datou_result : time spend for datou_step_exec : 34.749651193618774 time spend to save output : 4.614142179489136 total time spend for step 1 : 39.36379337310791 step2:crop_condition Tue Feb 4 11:03:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 20 ! batch 1 Loaded 138 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 20 About to insert : list_path_to_insert length 20 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1332639511, 1332639251, 1332639229, 1332639159, 1332638906, 1332638903, 1332638899, 1332638877, 1332638874, 1332638871, 1332638734, 1332638673, 1332638669, 1332638667, 1332638664, 1332638450, 1332638446, 1332638441, 1332638343, 1332638337] Looping around the photos to save general results len do output : 51 /-3653337379Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337384Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337382Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337397Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337443Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337457Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337476Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337490Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337503Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337513Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337512Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337394Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337431Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337462Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337471Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337439Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337441Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337451Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337459Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337480Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337498Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337390Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337409Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337414Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337416Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337430Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337442Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337437Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337445Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337460Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337455Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337458Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337470Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337468Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337469Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337477Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337474Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337488Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337492Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337495Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337501Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337434Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337446Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337453Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337466Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337510Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337508Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337514Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337438Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337456Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653337465Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639511', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639251', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639229', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639159', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638906', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638903', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638899', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638877', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638874', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638871', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638734', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638673', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638669', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638667', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638664', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638450', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638446', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638441', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638343', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638337', None, None, None, None, None, '2526005') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 173 time used for this insertion : 0.03715777397155762 save_final save missing photos in datou_result : time spend for datou_step_exec : 12.173872947692871 time spend to save output : 0.03853774070739746 total time spend for step 2 : 12.212410688400269 step3:thcl Tue Feb 4 11:03:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.005011320114135742 time to convert the images to numpy array : 4.291534423828125e-06 time to import caffe and check if the image exist : 0.005682706832885742 time to convert the images to numpy array : 0.031986236572265625 time to import caffe and check if the image exist : 0.007565975189208984 time to convert the images to numpy array : 0.029288768768310547 time to import caffe and check if the image exist : 0.010721683502197266 time to convert the images to numpy array : 0.030709505081176758 time to import caffe and check if the image exist : 0.009961605072021484 time to convert the images to numpy array : 0.037198781967163086 time to import caffe and check if the image exist : 0.008609533309936523 time to convert the images to numpy array : 0.04498934745788574 time to import caffe and check if the image exist : 0.007931947708129883 time to convert the images to numpy array : 0.05256795883178711 time to import caffe and check if the image exist : 0.012307882308959961 time to convert the images to numpy array : 0.04846906661987305 time to import caffe and check if the image exist : 0.011487007141113281 time to convert the images to numpy array : 0.050293922424316406 time to import caffe and check if the image exist : 0.0180511474609375 time to convert the images to numpy array : 0.04633164405822754 total time to convert the images to numpy array : 0.3554673194885254 list photo_ids error: [] list photo_ids correct : [-3653337379, -3653337384, -3653337382, -3653337397, -3653337443, -3653337457, -3653337446, -3653337453, -3653337466, -3653337510, -3653337508, -3653337514, -3653337438, -3653337456, -3653337465, -3653337459, -3653337480, -3653337498, -3653337390, -3653337409, -3653337414, -3653337455, -3653337458, -3653337470, -3653337468, -3653337469, -3653337477, -3653337431, -3653337462, -3653337471, -3653337439, -3653337441, -3653337451, -3653337416, -3653337430, -3653337442, -3653337437, -3653337445, -3653337460, -3653337474, -3653337488, -3653337492, -3653337495, -3653337501, -3653337434, -3653337476, -3653337490, -3653337503, -3653337513, -3653337512, -3653337394] number of photos to traite : 51 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9581 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5266 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.3865361213684082 time used to do the prediction : 0.18966007232666016 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5264 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5264 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.42227888107299805 time used to do the prediction : 0.1194005012512207 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.6450881958007812e-05 save missing photos in datou_result : time spend for datou_step_exec : 8.109814405441284 time spend to save output : 0.0005660057067871094 total time spend for step 3 : 8.110380411148071 step4:merge_mask_thcl_custom Tue Feb 4 11:03:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 138 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1332639511, 1332639251, 1332639229, 1332639159, 1332638906, 1332638903, 1332638899, 1332638877, 1332638874, 1332638871, 1332638734, 1332638673, 1332638669, 1332638667, 1332638664, 1332638450, 1332638446, 1332638441, 1332638343, 1332638337] Looping around the photos to save general results len do output : 20 /1332639511Didn't retrieve data .Didn't retrieve data . /1332639251Didn't retrieve data .Didn't retrieve data . /1332639229Didn't retrieve data .Didn't retrieve data . /1332639159Didn't retrieve data .Didn't retrieve data . /1332638906Didn't retrieve data .Didn't retrieve data . /1332638903Didn't retrieve data .Didn't retrieve data . /1332638899Didn't retrieve data .Didn't retrieve data . /1332638877Didn't retrieve data .Didn't retrieve data . /1332638874Didn't retrieve data .Didn't retrieve data . /1332638871Didn't retrieve data .Didn't retrieve data . /1332638734Didn't retrieve data .Didn't retrieve data . /1332638673Didn't retrieve data .Didn't retrieve data . /1332638669Didn't retrieve data .Didn't retrieve data . /1332638667Didn't retrieve data .Didn't retrieve data . /1332638664Didn't retrieve data .Didn't retrieve data . /1332638450Didn't retrieve data .Didn't retrieve data . /1332638446Didn't retrieve data .Didn't retrieve data . /1332638441Didn't retrieve data .Didn't retrieve data . /1332638343Didn't retrieve data .Didn't retrieve data . /1332638337Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639511', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639251', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639229', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639159', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638906', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638903', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638899', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638877', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638874', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638871', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638734', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638673', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638669', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638667', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638664', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638450', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638446', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638441', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638343', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638337', None, None, None, None, None, '2526005') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.02041459083557129 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.0650489330291748 time spend to save output : 0.021292448043823242 total time spend for step 4 : 0.08634138107299805 step5:rle_unique_nms_with_priority Tue Feb 4 11:03:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.3861701488494873 time for calcul the mask position with numpy : 0.017890214920043945 nb_pixel_total : 1058011 time to create 1 rle with new method : 0.04271507263183594 time for calcul the mask position with numpy : 0.006396293640136719 nb_pixel_total : 32461 time to create 1 rle with old method : 0.034702301025390625 time for calcul the mask position with numpy : 0.005983829498291016 nb_pixel_total : 2145 time to create 1 rle with old method : 0.0024755001068115234 time for calcul the mask position with numpy : 0.012380599975585938 nb_pixel_total : 926845 time to create 1 rle with new method : 0.04168963432312012 time for calcul the mask position with numpy : 0.006222248077392578 nb_pixel_total : 54138 time to create 1 rle with old method : 0.06253314018249512 create new chi : 0.2400343418121338 time to delete rle : 0.013702869415283203 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5776 TO DO : save crop sub photo not yet done ! save time : 0.3514978885650635 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 0.8409159183502197 time for calcul the mask position with numpy : 0.014619588851928711 nb_pixel_total : 1129240 time to create 1 rle with new method : 0.04182767868041992 time for calcul the mask position with numpy : 0.00604701042175293 nb_pixel_total : 6339 time to create 1 rle with old method : 0.007318735122680664 time for calcul the mask position with numpy : 0.0060045719146728516 nb_pixel_total : 1506 time to create 1 rle with old method : 0.0017023086547851562 time for calcul the mask position with numpy : 0.009871721267700195 nb_pixel_total : 585600 time to create 1 rle with new method : 0.03891587257385254 time for calcul the mask position with numpy : 0.0070688724517822266 nb_pixel_total : 30569 time to create 1 rle with old method : 0.03822684288024902 time for calcul the mask position with numpy : 0.0075032711029052734 nb_pixel_total : 219384 time to create 1 rle with new method : 0.029561519622802734 time for calcul the mask position with numpy : 0.0065746307373046875 nb_pixel_total : 4212 time to create 1 rle with old method : 0.0046062469482421875 time for calcul the mask position with numpy : 0.006436586380004883 nb_pixel_total : 8105 time to create 1 rle with old method : 0.009054422378540039 time for calcul the mask position with numpy : 0.006766319274902344 nb_pixel_total : 25300 time to create 1 rle with old method : 0.02795124053955078 time for calcul the mask position with numpy : 0.006898641586303711 nb_pixel_total : 63345 time to create 1 rle with old method : 0.07266020774841309 create new chi : 0.35440969467163086 time to delete rle : 0.0010485649108886719 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 6987 TO DO : save crop sub photo not yet done ! save time : 0.4335806369781494 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 0.6577751636505127 time for calcul the mask position with numpy : 0.013430118560791016 nb_pixel_total : 1043109 time to create 1 rle with new method : 0.03277587890625 time for calcul the mask position with numpy : 0.006911754608154297 nb_pixel_total : 91725 time to create 1 rle with old method : 0.09817647933959961 time for calcul the mask position with numpy : 0.00733184814453125 nb_pixel_total : 81335 time to create 1 rle with old method : 0.08782649040222168 time for calcul the mask position with numpy : 0.005952596664428711 nb_pixel_total : 13702 time to create 1 rle with old method : 0.015636682510375977 time for calcul the mask position with numpy : 0.011564493179321289 nb_pixel_total : 660648 time to create 1 rle with new method : 0.03374648094177246 time for calcul the mask position with numpy : 0.006188154220581055 nb_pixel_total : 3583 time to create 1 rle with old method : 0.004169464111328125 time for calcul the mask position with numpy : 0.008027315139770508 nb_pixel_total : 159630 time to create 1 rle with new method : 0.02985525131225586 time for calcul the mask position with numpy : 0.006339550018310547 nb_pixel_total : 19868 time to create 1 rle with old method : 0.022784709930419922 create new chi : 0.39408230781555176 time to delete rle : 0.0007071495056152344 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 7084 TO DO : save crop sub photo not yet done ! save time : 0.4243953227996826 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.5343232154846191 time for calcul the mask position with numpy : 0.017699718475341797 nb_pixel_total : 1152278 time to create 1 rle with new method : 0.036884307861328125 time for calcul the mask position with numpy : 0.006760597229003906 nb_pixel_total : 176311 time to create 1 rle with new method : 0.03162860870361328 time for calcul the mask position with numpy : 0.007427692413330078 nb_pixel_total : 227871 time to create 1 rle with new method : 0.028034210205078125 time for calcul the mask position with numpy : 0.010645866394042969 nb_pixel_total : 517140 time to create 1 rle with new method : 0.031820058822631836 create new chi : 0.17200613021850586 time to delete rle : 0.0009763240814208984 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 8066 TO DO : save crop sub photo not yet done ! save time : 0.47726941108703613 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 0.5744357109069824 time for calcul the mask position with numpy : 0.01548147201538086 nb_pixel_total : 934628 time to create 1 rle with new method : 0.045389652252197266 time for calcul the mask position with numpy : 0.008021831512451172 nb_pixel_total : 42116 time to create 1 rle with old method : 0.046193838119506836 time for calcul the mask position with numpy : 0.00658106803894043 nb_pixel_total : 32261 time to create 1 rle with old method : 0.03546261787414551 time for calcul the mask position with numpy : 0.0064220428466796875 nb_pixel_total : 23213 time to create 1 rle with old method : 0.02658224105834961 time for calcul the mask position with numpy : 0.007165670394897461 nb_pixel_total : 4574 time to create 1 rle with old method : 0.005636453628540039 time for calcul the mask position with numpy : 0.007514476776123047 nb_pixel_total : 31574 time to create 1 rle with old method : 0.03551030158996582 time for calcul the mask position with numpy : 0.014570951461791992 nb_pixel_total : 982984 time to create 1 rle with new method : 0.04549837112426758 time for calcul the mask position with numpy : 0.006960153579711914 nb_pixel_total : 22250 time to create 1 rle with old method : 0.028233766555786133 create new chi : 0.3499739170074463 time to delete rle : 0.0008115768432617188 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6818 TO DO : save crop sub photo not yet done ! save time : 0.39549756050109863 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.33975887298583984 time for calcul the mask position with numpy : 0.014314889907836914 nb_pixel_total : 1153021 time to create 1 rle with new method : 0.04171133041381836 time for calcul the mask position with numpy : 0.006216526031494141 nb_pixel_total : 5881 time to create 1 rle with old method : 0.006365060806274414 time for calcul the mask position with numpy : 0.0058171749114990234 nb_pixel_total : 11559 time to create 1 rle with old method : 0.012851238250732422 time for calcul the mask position with numpy : 0.0058863162994384766 nb_pixel_total : 2706 time to create 1 rle with old method : 0.003000497817993164 time for calcul the mask position with numpy : 0.006224632263183594 nb_pixel_total : 41177 time to create 1 rle with old method : 0.04492044448852539 time for calcul the mask position with numpy : 0.011809110641479492 nb_pixel_total : 859256 time to create 1 rle with new method : 0.03104686737060547 create new chi : 0.19571208953857422 time to delete rle : 0.0004792213439941406 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 4732 TO DO : save crop sub photo not yet done ! save time : 0.27523350715637207 nb_obj : 5 nb_hashtags : 1 time to prepare the origin masks : 0.36385154724121094 time for calcul the mask position with numpy : 0.016341447830200195 nb_pixel_total : 1140754 time to create 1 rle with new method : 0.05236339569091797 time for calcul the mask position with numpy : 0.007968664169311523 nb_pixel_total : 15227 time to create 1 rle with old method : 0.016943693161010742 time for calcul the mask position with numpy : 0.007669925689697266 nb_pixel_total : 105449 time to create 1 rle with old method : 0.11481738090515137 time for calcul the mask position with numpy : 0.011417627334594727 nb_pixel_total : 761956 time to create 1 rle with new method : 0.04487729072570801 time for calcul the mask position with numpy : 0.006625175476074219 nb_pixel_total : 30737 time to create 1 rle with old method : 0.03305864334106445 time for calcul the mask position with numpy : 0.006223917007446289 nb_pixel_total : 19477 time to create 1 rle with old method : 0.021641016006469727 create new chi : 0.35004258155822754 time to delete rle : 0.0005278587341308594 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5202 TO DO : save crop sub photo not yet done ! save time : 0.3197143077850342 nb_obj : 7 nb_hashtags : 1 time to prepare the origin masks : 0.6151344776153564 time for calcul the mask position with numpy : 0.0161893367767334 nb_pixel_total : 995877 time to create 1 rle with new method : 0.04604697227478027 time for calcul the mask position with numpy : 0.0061876773834228516 nb_pixel_total : 7614 time to create 1 rle with old method : 0.00822758674621582 time for calcul the mask position with numpy : 0.0069196224212646484 nb_pixel_total : 104285 time to create 1 rle with old method : 0.11566996574401855 time for calcul the mask position with numpy : 0.006520748138427734 nb_pixel_total : 79452 time to create 1 rle with old method : 0.08560776710510254 time for calcul the mask position with numpy : 0.006366252899169922 nb_pixel_total : 63431 time to create 1 rle with old method : 0.06871986389160156 time for calcul the mask position with numpy : 0.011647224426269531 nb_pixel_total : 792089 time to create 1 rle with new method : 0.0411221981048584 time for calcul the mask position with numpy : 0.006281137466430664 nb_pixel_total : 9160 time to create 1 rle with old method : 0.009893178939819336 time for calcul the mask position with numpy : 0.0061397552490234375 nb_pixel_total : 21692 time to create 1 rle with old method : 0.02346205711364746 create new chi : 0.47481536865234375 time to delete rle : 0.0006833076477050781 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 7247 TO DO : save crop sub photo not yet done ! save time : 0.4270803928375244 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.6563973426818848 time for calcul the mask position with numpy : 0.013687372207641602 nb_pixel_total : 905998 time to create 1 rle with new method : 0.04379749298095703 time for calcul the mask position with numpy : 0.006038665771484375 nb_pixel_total : 6899 time to create 1 rle with old method : 0.0074350833892822266 time for calcul the mask position with numpy : 0.00574493408203125 nb_pixel_total : 9205 time to create 1 rle with old method : 0.009735345840454102 time for calcul the mask position with numpy : 0.005880117416381836 nb_pixel_total : 38642 time to create 1 rle with old method : 0.04132080078125 time for calcul the mask position with numpy : 0.00764155387878418 nb_pixel_total : 366587 time to create 1 rle with new method : 0.03632688522338867 time for calcul the mask position with numpy : 0.010826826095581055 nb_pixel_total : 746269 time to create 1 rle with new method : 0.028657197952270508 create new chi : 0.2222919464111328 time to delete rle : 0.0007965564727783203 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 9359 TO DO : save crop sub photo not yet done ! save time : 0.5157208442687988 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 0.7036056518554688 time for calcul the mask position with numpy : 0.020035266876220703 nb_pixel_total : 1158711 time to create 1 rle with new method : 0.045807838439941406 time for calcul the mask position with numpy : 0.006201744079589844 nb_pixel_total : 95585 time to create 1 rle with old method : 0.09757661819458008 time for calcul the mask position with numpy : 0.006705045700073242 nb_pixel_total : 136717 time to create 1 rle with old method : 0.1415233612060547 time for calcul the mask position with numpy : 0.00609135627746582 nb_pixel_total : 460 time to create 1 rle with old method : 0.0007963180541992188 time for calcul the mask position with numpy : 0.005863189697265625 nb_pixel_total : 3492 time to create 1 rle with old method : 0.0039670467376708984 time for calcul the mask position with numpy : 0.009536266326904297 nb_pixel_total : 569292 time to create 1 rle with new method : 0.03928041458129883 time for calcul the mask position with numpy : 0.006570577621459961 nb_pixel_total : 4411 time to create 1 rle with old method : 0.007033348083496094 time for calcul the mask position with numpy : 0.006766557693481445 nb_pixel_total : 47481 time to create 1 rle with old method : 0.06381106376647949 time for calcul the mask position with numpy : 0.005865335464477539 nb_pixel_total : 2612 time to create 1 rle with old method : 0.0028524398803710938 time for calcul the mask position with numpy : 0.006049156188964844 nb_pixel_total : 54839 time to create 1 rle with old method : 0.05697751045227051 create new chi : 0.5447196960449219 time to delete rle : 0.0006372928619384766 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 6812 TO DO : save crop sub photo not yet done ! save time : 0.3839426040649414 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 0.4925525188446045 time for calcul the mask position with numpy : 0.017606258392333984 nb_pixel_total : 1224868 time to create 1 rle with new method : 0.03865528106689453 time for calcul the mask position with numpy : 0.0060214996337890625 nb_pixel_total : 2436 time to create 1 rle with old method : 0.0027256011962890625 time for calcul the mask position with numpy : 0.005853414535522461 nb_pixel_total : 8865 time to create 1 rle with old method : 0.009882450103759766 time for calcul the mask position with numpy : 0.007123470306396484 nb_pixel_total : 2878 time to create 1 rle with old method : 0.003314495086669922 time for calcul the mask position with numpy : 0.006720066070556641 nb_pixel_total : 96284 time to create 1 rle with old method : 0.10323190689086914 time for calcul the mask position with numpy : 0.006783485412597656 nb_pixel_total : 124653 time to create 1 rle with old method : 0.12939834594726562 time for calcul the mask position with numpy : 0.006200075149536133 nb_pixel_total : 15940 time to create 1 rle with old method : 0.017617464065551758 time for calcul the mask position with numpy : 0.006149768829345703 nb_pixel_total : 4982 time to create 1 rle with old method : 0.0057904720306396484 time for calcul the mask position with numpy : 0.00988149642944336 nb_pixel_total : 592694 time to create 1 rle with new method : 0.031096696853637695 create new chi : 0.417834997177124 time to delete rle : 0.0006339550018310547 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 6594 TO DO : save crop sub photo not yet done ! save time : 0.39727234840393066 nb_obj : 10 nb_hashtags : 4 time to prepare the origin masks : 1.0277464389801025 time for calcul the mask position with numpy : 0.0215301513671875 nb_pixel_total : 1706783 time to create 1 rle with new method : 0.04032588005065918 time for calcul the mask position with numpy : 0.0066623687744140625 nb_pixel_total : 11689 time to create 1 rle with old method : 0.013496160507202148 time for calcul the mask position with numpy : 0.00613856315612793 nb_pixel_total : 6217 time to create 1 rle with old method : 0.007411479949951172 time for calcul the mask position with numpy : 0.006819725036621094 nb_pixel_total : 87243 time to create 1 rle with old method : 0.09621739387512207 time for calcul the mask position with numpy : 0.006453514099121094 nb_pixel_total : 30057 time to create 1 rle with old method : 0.03397083282470703 time for calcul the mask position with numpy : 0.006395578384399414 nb_pixel_total : 708 time to create 1 rle with old method : 0.001146078109741211 time for calcul the mask position with numpy : 0.007115364074707031 nb_pixel_total : 703 time to create 1 rle with old method : 0.0010881423950195312 time for calcul the mask position with numpy : 0.006936311721801758 nb_pixel_total : 144950 time to create 1 rle with old method : 0.1566150188446045 time for calcul the mask position with numpy : 0.006146669387817383 nb_pixel_total : 43966 time to create 1 rle with old method : 0.0481257438659668 time for calcul the mask position with numpy : 0.0066301822662353516 nb_pixel_total : 26779 time to create 1 rle with old method : 0.03023672103881836 time for calcul the mask position with numpy : 0.00632786750793457 nb_pixel_total : 14505 time to create 1 rle with old method : 0.015542984008789062 create new chi : 0.5351691246032715 time to delete rle : 0.0006103515625 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 6474 TO DO : save crop sub photo not yet done ! save time : 0.42131590843200684 nb_obj : 9 nb_hashtags : 5 time to prepare the origin masks : 0.9354653358459473 time for calcul the mask position with numpy : 0.0150604248046875 nb_pixel_total : 869002 time to create 1 rle with new method : 0.04507946968078613 time for calcul the mask position with numpy : 0.006050586700439453 nb_pixel_total : 9355 time to create 1 rle with old method : 0.01037740707397461 time for calcul the mask position with numpy : 0.006019115447998047 nb_pixel_total : 879 time to create 1 rle with old method : 0.001055002212524414 time for calcul the mask position with numpy : 0.00592803955078125 nb_pixel_total : 37242 time to create 1 rle with old method : 0.039108991622924805 time for calcul the mask position with numpy : 0.006064176559448242 nb_pixel_total : 35083 time to create 1 rle with old method : 0.03727364540100098 time for calcul the mask position with numpy : 0.006774187088012695 nb_pixel_total : 77230 time to create 1 rle with old method : 0.08145523071289062 time for calcul the mask position with numpy : 0.006843090057373047 nb_pixel_total : 137370 time to create 1 rle with old method : 0.14224815368652344 time for calcul the mask position with numpy : 0.0062329769134521484 nb_pixel_total : 1060 time to create 1 rle with old method : 0.001505136489868164 time for calcul the mask position with numpy : 0.005991220474243164 nb_pixel_total : 41326 time to create 1 rle with old method : 0.041791439056396484 time for calcul the mask position with numpy : 0.01186823844909668 nb_pixel_total : 865053 time to create 1 rle with new method : 0.041359663009643555 create new chi : 0.5278139114379883 time to delete rle : 0.00072479248046875 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 7992 TO DO : save crop sub photo not yet done ! save time : 0.4655599594116211 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 0.41554999351501465 time for calcul the mask position with numpy : 0.015321731567382812 nb_pixel_total : 1499324 time to create 1 rle with new method : 0.03033280372619629 time for calcul the mask position with numpy : 0.006656646728515625 nb_pixel_total : 45091 time to create 1 rle with old method : 0.04715323448181152 time for calcul the mask position with numpy : 0.0060350894927978516 nb_pixel_total : 6415 time to create 1 rle with old method : 0.007088899612426758 time for calcul the mask position with numpy : 0.0058672428131103516 nb_pixel_total : 9460 time to create 1 rle with old method : 0.010299444198608398 time for calcul the mask position with numpy : 0.006011486053466797 nb_pixel_total : 12812 time to create 1 rle with old method : 0.014019966125488281 time for calcul the mask position with numpy : 0.005861043930053711 nb_pixel_total : 8325 time to create 1 rle with old method : 0.009100675582885742 time for calcul the mask position with numpy : 0.008750677108764648 nb_pixel_total : 492173 time to create 1 rle with new method : 0.026946306228637695 create new chi : 0.2000749111175537 time to delete rle : 0.0004456043243408203 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 4468 TO DO : save crop sub photo not yet done ! save time : 0.26693153381347656 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.34000658988952637 time for calcul the mask position with numpy : 0.015301704406738281 nb_pixel_total : 1186572 time to create 1 rle with new method : 0.04271435737609863 time for calcul the mask position with numpy : 0.0061228275299072266 nb_pixel_total : 7288 time to create 1 rle with old method : 0.008032560348510742 time for calcul the mask position with numpy : 0.006537914276123047 nb_pixel_total : 91418 time to create 1 rle with old method : 0.09530377388000488 time for calcul the mask position with numpy : 0.011255025863647461 nb_pixel_total : 721858 time to create 1 rle with new method : 0.043166399002075195 time for calcul the mask position with numpy : 0.006561279296875 nb_pixel_total : 61680 time to create 1 rle with old method : 0.06509232521057129 time for calcul the mask position with numpy : 0.0060503482818603516 nb_pixel_total : 4784 time to create 1 rle with old method : 0.00508570671081543 create new chi : 0.3208434581756592 time to delete rle : 0.0006625652313232422 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6551 TO DO : save crop sub photo not yet done ! save time : 0.3966083526611328 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 0.9659016132354736 time for calcul the mask position with numpy : 0.025348424911499023 nb_pixel_total : 1290454 time to create 1 rle with new method : 0.05052375793457031 time for calcul the mask position with numpy : 0.007167339324951172 nb_pixel_total : 48974 time to create 1 rle with old method : 0.05422782897949219 time for calcul the mask position with numpy : 0.007402658462524414 nb_pixel_total : 7761 time to create 1 rle with old method : 0.00864553451538086 time for calcul the mask position with numpy : 0.00620269775390625 nb_pixel_total : 25900 time to create 1 rle with old method : 0.02778792381286621 time for calcul the mask position with numpy : 0.006291389465332031 nb_pixel_total : 6659 time to create 1 rle with old method : 0.0076770782470703125 time for calcul the mask position with numpy : 0.005842447280883789 nb_pixel_total : 1555 time to create 1 rle with old method : 0.0016484260559082031 time for calcul the mask position with numpy : 0.010788679122924805 nb_pixel_total : 644854 time to create 1 rle with new method : 0.03813600540161133 time for calcul the mask position with numpy : 0.0063648223876953125 nb_pixel_total : 5377 time to create 1 rle with old method : 0.0059206485748291016 time for calcul the mask position with numpy : 0.0062046051025390625 nb_pixel_total : 4684 time to create 1 rle with old method : 0.0049555301666259766 time for calcul the mask position with numpy : 0.0067403316497802734 nb_pixel_total : 37382 time to create 1 rle with old method : 0.040490150451660156 create new chi : 0.33409833908081055 time to delete rle : 0.0007567405700683594 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 5940 TO DO : save crop sub photo not yet done ! save time : 0.3485422134399414 nb_obj : 4 nb_hashtags : 3 time to prepare the origin masks : 0.28113317489624023 time for calcul the mask position with numpy : 0.016771316528320312 nb_pixel_total : 1625145 time to create 1 rle with new method : 0.0291290283203125 time for calcul the mask position with numpy : 0.006278514862060547 nb_pixel_total : 32813 time to create 1 rle with old method : 0.03533339500427246 time for calcul the mask position with numpy : 0.008265018463134766 nb_pixel_total : 396076 time to create 1 rle with new method : 0.0268552303314209 time for calcul the mask position with numpy : 0.005823373794555664 nb_pixel_total : 5965 time to create 1 rle with old method : 0.006404399871826172 time for calcul the mask position with numpy : 0.005836963653564453 nb_pixel_total : 13601 time to create 1 rle with old method : 0.014679908752441406 create new chi : 0.15853214263916016 time to delete rle : 0.0003848075866699219 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4080 TO DO : save crop sub photo not yet done ! save time : 0.2768542766571045 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.43487048149108887 time for calcul the mask position with numpy : 0.015338420867919922 nb_pixel_total : 1374964 time to create 1 rle with new method : 0.03296661376953125 time for calcul the mask position with numpy : 0.0071566104888916016 nb_pixel_total : 200513 time to create 1 rle with new method : 0.02836132049560547 time for calcul the mask position with numpy : 0.006293773651123047 nb_pixel_total : 54865 time to create 1 rle with old method : 0.06000065803527832 time for calcul the mask position with numpy : 0.006333589553833008 nb_pixel_total : 49966 time to create 1 rle with old method : 0.05559945106506348 time for calcul the mask position with numpy : 0.006285667419433594 nb_pixel_total : 46387 time to create 1 rle with old method : 0.05309629440307617 time for calcul the mask position with numpy : 0.00891423225402832 nb_pixel_total : 346905 time to create 1 rle with new method : 0.028485536575317383 create new chi : 0.309842586517334 time to delete rle : 0.0007009506225585938 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 7566 TO DO : save crop sub photo not yet done ! save time : 0.42991042137145996 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.34287214279174805 time for calcul the mask position with numpy : 0.021981000900268555 nb_pixel_total : 1590728 time to create 1 rle with new method : 0.04691481590270996 time for calcul the mask position with numpy : 0.010424137115478516 nb_pixel_total : 407035 time to create 1 rle with new method : 0.04226350784301758 time for calcul the mask position with numpy : 0.008037567138671875 nb_pixel_total : 19846 time to create 1 rle with old method : 0.021726608276367188 time for calcul the mask position with numpy : 0.008129119873046875 nb_pixel_total : 55991 time to create 1 rle with old method : 0.07252097129821777 create new chi : 0.23699474334716797 time to delete rle : 0.0007557868957519531 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5544 TO DO : save crop sub photo not yet done ! save time : 0.3275322914123535 nb_obj : 7 nb_hashtags : 4 time to prepare the origin masks : 0.7526135444641113 time for calcul the mask position with numpy : 0.025176525115966797 nb_pixel_total : 1536568 time to create 1 rle with new method : 0.04623842239379883 time for calcul the mask position with numpy : 0.00686955451965332 nb_pixel_total : 4772 time to create 1 rle with old method : 0.0051116943359375 time for calcul the mask position with numpy : 0.006674766540527344 nb_pixel_total : 8155 time to create 1 rle with old method : 0.00879812240600586 time for calcul the mask position with numpy : 0.006794929504394531 nb_pixel_total : 11538 time to create 1 rle with old method : 0.012077808380126953 time for calcul the mask position with numpy : 0.007462263107299805 nb_pixel_total : 14486 time to create 1 rle with old method : 0.01538395881652832 time for calcul the mask position with numpy : 0.010117053985595703 nb_pixel_total : 451062 time to create 1 rle with new method : 0.041521310806274414 time for calcul the mask position with numpy : 0.007268667221069336 nb_pixel_total : 33955 time to create 1 rle with old method : 0.03782057762145996 time for calcul the mask position with numpy : 0.006560087203979492 nb_pixel_total : 13064 time to create 1 rle with old method : 0.014048337936401367 create new chi : 0.2614309787750244 time to delete rle : 0.0008718967437744141 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 7802 TO DO : save crop sub photo not yet done ! save time : 0.41738438606262207 map_output_result : {1332639511: (0.0, 'Should be the crop_list due to order', 0.0), 1332639251: (0.0, 'Should be the crop_list due to order', 0.0), 1332639229: (0.0, 'Should be the crop_list due to order', 0.0), 1332639159: (0.0, 'Should be the crop_list due to order', 0.0), 1332638906: (0.0, 'Should be the crop_list due to order', 0.0), 1332638903: (0.0, 'Should be the crop_list due to order', 0.0), 1332638899: (0.0, 'Should be the crop_list due to order', 0.0), 1332638877: (0.0, 'Should be the crop_list due to order', 0.0), 1332638874: (0.0, 'Should be the crop_list due to order', 0.0), 1332638871: (0.0, 'Should be the crop_list due to order', 0.0), 1332638734: (0.0, 'Should be the crop_list due to order', 0.0), 1332638673: (0.0, 'Should be the crop_list due to order', 0.0), 1332638669: (0.0, 'Should be the crop_list due to order', 0.0), 1332638667: (0.0, 'Should be the crop_list due to order', 0.0), 1332638664: (0.0, 'Should be the crop_list due to order', 0.0), 1332638450: (0.0, 'Should be the crop_list due to order', 0.0), 1332638446: (0.0, 'Should be the crop_list due to order', 0.0), 1332638441: (0.0, 'Should be the crop_list due to order', 0.0), 1332638343: (0.0, 'Should be the crop_list due to order', 0.0), 1332638337: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1332639511, 1332639251, 1332639229, 1332639159, 1332638906, 1332638903, 1332638899, 1332638877, 1332638874, 1332638871, 1332638734, 1332638673, 1332638669, 1332638667, 1332638664, 1332638450, 1332638446, 1332638441, 1332638343, 1332638337] Looping around the photos to save general results len do output : 20 /1332639511.Didn't retrieve data . /1332639251.Didn't retrieve data . /1332639229.Didn't retrieve data . /1332639159.Didn't retrieve data . /1332638906.Didn't retrieve data . /1332638903.Didn't retrieve data . /1332638899.Didn't retrieve data . /1332638877.Didn't retrieve data . /1332638874.Didn't retrieve data . /1332638871.Didn't retrieve data . /1332638734.Didn't retrieve data . /1332638673.Didn't retrieve data . /1332638669.Didn't retrieve data . /1332638667.Didn't retrieve data . /1332638664.Didn't retrieve data . /1332638450.Didn't retrieve data . /1332638446.Didn't retrieve data . /1332638441.Didn't retrieve data . /1332638343.Didn't retrieve data . /1332638337.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639511', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639251', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639229', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639159', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638906', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638903', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638899', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638877', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638874', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638871', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638734', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638673', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638669', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638667', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638664', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638450', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638446', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638441', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638343', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638337', None, None, None, None, None, '2526005') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.014383554458618164 save_final save missing photos in datou_result : time spend for datou_step_exec : 26.637688398361206 time spend to save output : 0.01513814926147461 total time spend for step 5 : 26.65282654762268 step6:crop_condition Tue Feb 4 11:04:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 20 ! batch 1 Loaded 147 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663448_3586921 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8527863025665283 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663451_3586921 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0261363983154297 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663453_3586921 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5783023834228516 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663455_3586921 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.6100809574127197 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663459_3586921 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.0721426010131836 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663462_3586921 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7292637825012207 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663464_3586921 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7456803321838379 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1332639511, 1332639251, 1332639229, 1332639159, 1332638906, 1332638903, 1332638899, 1332638877, 1332638874, 1332638871, 1332638734, 1332638673, 1332638669, 1332638667, 1332638664, 1332638450, 1332638446, 1332638441, 1332638343, 1332638337] Looping around the photos to save general results len do output : 26 /1334520676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520677Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520678Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520680Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520681Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520682Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520683Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520684Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520686Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520688Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520691Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520697Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520698Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520699Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520705Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520706Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520707Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520708Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520716Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520719Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639511', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639251', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639229', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639159', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638906', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638903', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638899', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638877', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638874', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638871', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638734', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638673', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638669', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638667', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638664', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638450', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638446', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638441', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638343', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638337', None, None, None, None, None, '2526005') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 98 time used for this insertion : 0.029674530029296875 save_final save missing photos in datou_result : time spend for datou_step_exec : 18.968931913375854 time spend to save output : 0.03087782859802246 total time spend for step 6 : 18.999809741973877 step7:ventilate_hashtags_in_portfolio Tue Feb 4 11:04:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20043199 get user id for portfolio 20043199 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043199 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','environnement','plastique','autre_refus','kraft','papier','flou','cartonnette','mal_croppe','Teint_Dans_La_Masse','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043199 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','environnement','plastique','autre_refus','kraft','papier','flou','cartonnette','mal_croppe','Teint_Dans_La_Masse','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043199 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','environnement','plastique','autre_refus','kraft','papier','flou','cartonnette','mal_croppe','Teint_Dans_La_Masse','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20206430,20206431,20206432,20206433,20206434,20206435,20206436,20206437,20206438,20206439,20206440,20206441?tags=metal,environnement,plastique,autre_refus,kraft,papier,flou,cartonnette,mal_croppe,Teint_Dans_La_Masse,Carton_brun,Carton_gris Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1332639511, 1332639251, 1332639229, 1332639159, 1332638906, 1332638903, 1332638899, 1332638877, 1332638874, 1332638871, 1332638734, 1332638673, 1332638669, 1332638667, 1332638664, 1332638450, 1332638446, 1332638441, 1332638343, 1332638337] Looping around the photos to save general results len do output : 1 /20043199. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639511', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639251', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639229', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639159', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638906', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638903', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638899', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638877', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638874', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638871', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638734', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638673', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638669', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638667', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638664', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638450', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638446', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638441', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638343', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638337', None, None, None, None, None, '2526005') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.14034414291381836 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.7572975158691406 time spend to save output : 0.14079546928405762 total time spend for step 7 : 0.8980929851531982 step8:final Tue Feb 4 11:04:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1332639511: ('0.04420808919501898',), 1332639251: ('0.04420808919501898',), 1332639229: ('0.04420808919501898',), 1332639159: ('0.04420808919501898',), 1332638906: ('0.04420808919501898',), 1332638903: ('0.04420808919501898',), 1332638899: ('0.04420808919501898',), 1332638877: ('0.04420808919501898',), 1332638874: ('0.04420808919501898',), 1332638871: ('0.04420808919501898',), 1332638734: ('0.04420808919501898',), 1332638673: ('0.04420808919501898',), 1332638669: ('0.04420808919501898',), 1332638667: ('0.04420808919501898',), 1332638664: ('0.04420808919501898',), 1332638450: ('0.04420808919501898',), 1332638446: ('0.04420808919501898',), 1332638441: ('0.04420808919501898',), 1332638343: ('0.04420808919501898',), 1332638337: ('0.04420808919501898',)} new output for save of step final : {1332639511: ('0.04420808919501898',), 1332639251: ('0.04420808919501898',), 1332639229: ('0.04420808919501898',), 1332639159: ('0.04420808919501898',), 1332638906: ('0.04420808919501898',), 1332638903: ('0.04420808919501898',), 1332638899: ('0.04420808919501898',), 1332638877: ('0.04420808919501898',), 1332638874: ('0.04420808919501898',), 1332638871: ('0.04420808919501898',), 1332638734: ('0.04420808919501898',), 1332638673: ('0.04420808919501898',), 1332638669: ('0.04420808919501898',), 1332638667: ('0.04420808919501898',), 1332638664: ('0.04420808919501898',), 1332638450: ('0.04420808919501898',), 1332638446: ('0.04420808919501898',), 1332638441: ('0.04420808919501898',), 1332638343: ('0.04420808919501898',), 1332638337: ('0.04420808919501898',)} [1332639511, 1332639251, 1332639229, 1332639159, 1332638906, 1332638903, 1332638899, 1332638877, 1332638874, 1332638871, 1332638734, 1332638673, 1332638669, 1332638667, 1332638664, 1332638450, 1332638446, 1332638441, 1332638343, 1332638337] Looping around the photos to save general results len do output : 20 /1332639511.Didn't retrieve data . /1332639251.Didn't retrieve data . /1332639229.Didn't retrieve data . /1332639159.Didn't retrieve data . /1332638906.Didn't retrieve data . /1332638903.Didn't retrieve data . /1332638899.Didn't retrieve data . /1332638877.Didn't retrieve data . /1332638874.Didn't retrieve data . /1332638871.Didn't retrieve data . /1332638734.Didn't retrieve data . /1332638673.Didn't retrieve data . /1332638669.Didn't retrieve data . /1332638667.Didn't retrieve data . /1332638664.Didn't retrieve data . /1332638450.Didn't retrieve data . /1332638446.Didn't retrieve data . /1332638441.Didn't retrieve data . /1332638343.Didn't retrieve data . /1332638337.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639511', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639251', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639229', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639159', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638906', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638903', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638899', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638877', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638874', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638871', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638734', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638673', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638669', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638667', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638664', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638450', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638446', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638441', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638343', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638337', None, None, None, None, None, '2526005') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.015186548233032227 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.13869595527648926 time spend to save output : 0.016211748123168945 total time spend for step 8 : 0.1549077033996582 step9:velours_tree Tue Feb 4 11:04:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.0617218017578125 time spend to save output : 4.506111145019531e-05 total time spend for step 9 : 0.061766862869262695 step10:send_mail_cod Tue Feb 4 11:04:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin in order to get the selector url, please entre the license of selector results_Auto_P20043199_04-02-2025_11_04_26.pdf 20206430 change filename to text .change filename to text .imagette202064301738663466 20206432 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064321738663466 20206433 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064331738663468 20206434 change filename to text .change filename to text .imagette202064341738663469 20206436 imagette202064361738663469 20206437 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064371738663469 20206438 imagette202064381738663471 20206439 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064391738663471 20206440 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064401738663473 20206441 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064411738663474 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20043199 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20206430,20206431,20206432,20206433,20206434,20206435,20206436,20206437,20206438,20206439,20206440,20206441?tags=metal,environnement,plastique,autre_refus,kraft,papier,flou,cartonnette,mal_croppe,Teint_Dans_La_Masse,Carton_brun,Carton_gris your option no_mail is active, we will not send the real mail to your client args[1332639511] : ('0.04420808919501898',) no score found for photo 1332639511 We are sending mail with results at report@fotonower.com args[1332639251] : ('0.04420808919501898',) no score found for photo 1332639251 We are sending mail with results at report@fotonower.com args[1332639229] : ('0.04420808919501898',) no score found for photo 1332639229 We are sending mail with results at report@fotonower.com args[1332639159] : ('0.04420808919501898',) no score found for photo 1332639159 We are sending mail with results at report@fotonower.com args[1332638906] : ('0.04420808919501898',) no score found for photo 1332638906 We are sending mail with results at report@fotonower.com args[1332638903] : ('0.04420808919501898',) no score found for photo 1332638903 We are sending mail with results at report@fotonower.com args[1332638899] : ('0.04420808919501898',) no score found for photo 1332638899 We are sending mail with results at report@fotonower.com args[1332638877] : ('0.04420808919501898',) no score found for photo 1332638877 We are sending mail with results at report@fotonower.com args[1332638874] : ('0.04420808919501898',) no score found for photo 1332638874 We are sending mail with results at report@fotonower.com args[1332638871] : ('0.04420808919501898',) no score found for photo 1332638871 We are sending mail with results at report@fotonower.com args[1332638734] : ('0.04420808919501898',) no score found for photo 1332638734 We are sending mail with results at report@fotonower.com args[1332638673] : ('0.04420808919501898',) no score found for photo 1332638673 We are sending mail with results at report@fotonower.com args[1332638669] : ('0.04420808919501898',) no score found for photo 1332638669 We are sending mail with results at report@fotonower.com args[1332638667] : ('0.04420808919501898',) no score found for photo 1332638667 We are sending mail with results at report@fotonower.com args[1332638664] : ('0.04420808919501898',) no score found for photo 1332638664 We are sending mail with results at report@fotonower.com args[1332638450] : ('0.04420808919501898',) no score found for photo 1332638450 We are sending mail with results at report@fotonower.com args[1332638446] : ('0.04420808919501898',) no score found for photo 1332638446 We are sending mail with results at report@fotonower.com args[1332638441] : ('0.04420808919501898',) no score found for photo 1332638441 We are sending mail with results at report@fotonower.com args[1332638343] : ('0.04420808919501898',) no score found for photo 1332638343 We are sending mail with results at report@fotonower.com args[1332638337] : ('0.04420808919501898',) no score found for photo 1332638337 We are sending mail with results at report@fotonower.com refus_total : 0.04420808919501898 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20043199 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1332637864,1332586371,1332638199,1332586379,1332638343,1332638441,1332586399,1332638446,1332638450,1332638165,1332638160,1332638673,1332586997,1332588296,1332588163,1332588259,1332637998,1332587577,1332585415,1332637877) Found this number of photos: 20 begin to download photo : 1332637864 begin to download photo : 1332638441 begin to download photo : 1332638160 begin to download photo : 1332588259 download finish for photo 1332588259 begin to download photo : 1332637998 download finish for photo 1332637864 begin to download photo : 1332586371 download finish for photo 1332638441 begin to download photo : 1332586399 download finish for photo 1332638160 begin to download photo : 1332638673 download finish for photo 1332637998 begin to download photo : 1332587577 download finish for photo 1332586399 begin to download photo : 1332638446 download finish for photo 1332586371 begin to download photo : 1332638199 download finish for photo 1332638673 begin to download photo : 1332586997 download finish for photo 1332638446 begin to download photo : 1332638450 download finish for photo 1332587577 begin to download photo : 1332585415 download finish for photo 1332638199 begin to download photo : 1332586379 download finish for photo 1332586997 begin to download photo : 1332588296 download finish for photo 1332638450 begin to download photo : 1332638165 download finish for photo 1332585415 begin to download photo : 1332637877 download finish for photo 1332586379 begin to download photo : 1332638343 download finish for photo 1332638165 download finish for photo 1332638343 download finish for photo 1332588296 begin to download photo : 1332588163 download finish for photo 1332637877 download finish for photo 1332588163 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043199_04-02-2025_11_04_26.pdf results_Auto_P20043199_04-02-2025_11_04_26.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043199_04-02-2025_11_04_26.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20043199','results_Auto_P20043199_04-02-2025_11_04_26.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043199_04-02-2025_11_04_26.pdf','pdf','','0.64','0.04420808919501898') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1332639511, 1332639251, 1332639229, 1332639159, 1332638906, 1332638903, 1332638899, 1332638877, 1332638874, 1332638871, 1332638734, 1332638673, 1332638669, 1332638667, 1332638664, 1332638450, 1332638446, 1332638441, 1332638343, 1332638337] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639511', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639251', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639229', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639159', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638906', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638903', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638899', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638877', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638874', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638871', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638734', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638673', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638669', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638667', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638664', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638450', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638446', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638441', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638343', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638337', None, None, None, None, None, '2526005') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 20 time used for this insertion : 0.01952362060546875 save_final save missing photos in datou_result : time spend for datou_step_exec : 11.372032642364502 time spend to save output : 0.020027875900268555 total time spend for step 10 : 11.39206051826477 step11:split_time_score Tue Feb 4 11:04:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 199),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 29012025 20043199 Nombre de photos uploadées : 199 / 23040 (0%) 29012025 20043199 Nombre de photos taguées (types de déchets): 0 / 199 (0%) 29012025 20043199 Nombre de photos taguées (volume) : 0 / 199 (0%) elapsed_time : load_data_split_time_score 6.198883056640625e-06 elapsed_time : order_list_meta_photo_and_scores 1.6689300537109375e-05 ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.00808262825012207 elapsed_time : insert_dashboard_record_day_entry 0.02220320701599121 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.05964569986773992 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20038647_29-01-2025_09_33_49.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20038647 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20038647 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20038303 order by id desc limit 1 Qualite : 0.05453990257520215 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20036989_29-01-2025_08_12_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20036989 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20036989 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20037835 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20037561 order by id desc limit 1 Qualite : 0.029203380749725394 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20037563_29-01-2025_08_42_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20037563 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20037563 AND mptpi.`type`=3726 To do Qualite : 0.035880779652306634 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20038304_29-01-2025_09_22_44.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20038304 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20038304 AND mptpi.`type`=3726 To do Qualite : 0.04420808919501898 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043199_04-02-2025_11_04_26.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043199 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043199 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20040004 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20040005 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042811 order by id desc limit 1 Qualite : 0.04179473612560559 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20042812_29-01-2025_13_32_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042812 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20042812 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042942 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043200 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042552 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042169 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042170 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042172 order by id desc limit 1 Qualite : 0.03250609913107722 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043201_29-01-2025_14_11_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043201 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043201 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043550 order by id desc limit 1 Qualite : 0.009347489222823262 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043551_29-01-2025_14_41_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043551 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043551 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043552 order by id desc limit 1 Qualite : 0.040548452513978116 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20044107_29-01-2025_15_32_02.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20044107 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20044107 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20044108 order by id desc limit 1 Qualite : 0.04088001272734784 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20044298_29-01-2025_15_27_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20044298 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20044298 AND mptpi.`type`=3726 To do Qualite : 0.025866735673049823 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20044671_29-01-2025_16_01_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20044671 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20044671 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045174 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045175 order by id desc limit 1 Qualite : 0.036707148122498 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20045657_29-01-2025_19_01_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045657 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20045657 AND mptpi.`type`=3726 To do Qualite : 0.04449978208679873 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20045658_29-01-2025_16_51_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045658 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20045658 AND mptpi.`type`=3726 To do Qualite : 0.06389528464627381 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20045933_29-01-2025_17_33_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045933 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20045933 AND mptpi.`type`=3726 To do Qualite : 0.05614592496812481 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20046299_29-01-2025_18_02_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20046299 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20046299 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20047762 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20047763 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20047764 order by id desc limit 1 Qualite : 0.0513536384207915 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20047765_29-01-2025_19_33_11.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20047765 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20047765 AND mptpi.`type`=3726 To do Qualite : 0.030150108126073005 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20048159_29-01-2025_20_01_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048159 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20048159 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048160 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048632 order by id desc limit 1 Qualite : 0.02217739158266012 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20048633_29-01-2025_20_21_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048633 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20048633 AND mptpi.`type`=3726 To do Qualite : 0.04372363561039585 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20048832_29-01-2025_20_27_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048832 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20048832 AND mptpi.`type`=3726 To do Qualite : 0.007334799965609816 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20049145_29-01-2025_21_00_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20049145 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20049145 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20105092 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066611 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066612 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066613 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066615 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066616 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066617 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20105093 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066619 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066620 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066621 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20105118 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'29012025': {'nb_upload': 199, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1332639511, 1332639251, 1332639229, 1332639159, 1332638906, 1332638903, 1332638899, 1332638877, 1332638874, 1332638871, 1332638734, 1332638673, 1332638669, 1332638667, 1332638664, 1332638450, 1332638446, 1332638441, 1332638343, 1332638337] Looping around the photos to save general results len do output : 1 /20043199Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639511', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639251', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639229', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332639159', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638906', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638903', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638899', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638877', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638874', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638871', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638734', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638673', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638669', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638667', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638664', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638450', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638446', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638441', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638343', None, None, None, None, None, '2526005') ('3459', None, None, None, None, None, None, None, '2526005') ('3459', '20043199', '1332638337', None, None, None, None, None, '2526005') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.015729188919067383 save_final save missing photos in datou_result : time spend for datou_step_exec : 29.482974767684937 time spend to save output : 0.01629805564880371 total time spend for step 11 : 29.49927282333374 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 74.75user 24.89system 2:32.40elapsed 65%CPU (0avgtext+0avgdata 3086672maxresident)k 1264inputs+65960outputs (11major+2097287minor)pagefaults 0swaps